Projection Casting in Virtual Environments

ABSTRACT

The present technology relates to artificial reality systems. Such systems provide projections a user can create to specify object interactions. For example, when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects. The present technology further includes a model for differentiating between global and local modes for, e.g., providing different input modalities or interpretations of user gestures.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.17/211,771, filed Mar. 24, 2021, currently pending, which is acontinuation of U.S. patent application Ser. No. 16/583,069 filed Sep.25, 2019, now U.S. Pat. No. 10,991,163 issued Apr. 27, 2021, which is acontinuation-in-part of U.S. patent application Ser. No. 16/578,221filed Sep. 20, 2019, now U.S. Pat. No. 11,176,745 issued Nov. 16, 2021.which are herein incorporated by reference in their entireties. Thisapplication is also related to U.S. patent application Ser. No.16/578,236 filed Sep. 20, 2019, now U.S. Pat. No. 11,189,099 issued Nov.30, 2021, U.S. patent application Ser. No. 16/578,240 filed Sep. 20,2019, now issued U.S. Pat. No. 11,086,406 issued Aug. 10, 2021, U.S.patent application Ser. No. 16/578,251 filed Sep. 20, 2019, now issuedU.S. Pat. No. 11,170,576 issued Nov. 9, 2021, and U.S. patentapplication Ser. No. 16/578,260 filed Sep. 20, 2019, now U.S. Pat. No.10,802,600, issued Oct. 13, 2020, all of which are herein incorporatedby reference in their entireties.

TECHNICAL FIELD

The present disclosure is directed to object interaction in anartificial reality environment.

BACKGROUND

In an artificial reality environment, some or all of the objects a usersees and interacts with are “virtual objects,” i.e., representations ofobjects generated by a computing system that appear in an environment.Virtual objects in an artificial reality environment can be presented tothe user by a head-mounted display, a mobile device, a projectionsystem, or another computing system. Often, users can interact withvirtual objects using controllers and/or gestures. For example, user“interactions” with virtual objects can include selecting, moving,rotating, resizing, actuating controls, changing colors or skins,defining interactions between real or virtual objects, setting virtualforces to act on virtual objects, or any other action on or change to anobject that a user can imagine. In some systems, a user can alsointeract with “real objects” that exist independent of the computersystem controlling the artificial reality environment. For example, auser can select a real object and add a virtual overlay to change theobject's color or some other way the object is presented to the user,cause it to affect other virtual objects, etc. As used herein, unlessotherwise specified, an “object” can be a real or virtual object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an overview of devices on whichsome implementations of the present technology can operate.

FIG. 2A is a wire diagram illustrating a virtual reality headset whichcan be used in some implementations of the present technology.

FIG. 2B is a wire diagram illustrating a mixed reality headset which canbe used in some implementations of the present technology.

FIG. 3 is a block diagram illustrating an overview of an environment inwhich some implementations of the present technology can operate.

FIG. 4 is a block diagram illustrating components which, in someimplementations, can be used in a system employing the disclosedtechnology.

FIG. 5 is a flow diagram illustrating a process used in someimplementations of the present technology for casting a projection withan origin point and a control point.

FIG. 6 is a conceptual diagram illustrating an example ray projectionwith a shoulder origin point and palm control point.

FIG. 7 is a conceptual diagram illustrating an example ray projectionwith a dominant eye origin point and a fingertip control point.

FIG. 8 is a conceptual diagram illustrating an example ray projectionwith a hip origin point and a fist control point.

FIG. 9 is a conceptual diagram illustrating an example ray projectionwith a shoulder origin point and a wrist control point.

FIG. 10 is a flow diagram illustrating a process used in someimplementations of the present technology for adjusting a projectionorigin point.

FIGS. 11A, 11B and 11C are conceptual diagrams illustrating an exampleof adjusting a projection origin between a hip and a shoulder.

FIG. 12 is a flow diagram illustrating a process used in someimplementations of the present technology for object interaction with aray projection.

FIG. 13 is a conceptual diagram illustrating an example ray projection.

FIG. 14 is a flow diagram illustrating a process used in someimplementations of the present technology for object interaction with acylinder projection.

FIG. 15 is a conceptual diagram illustrating an example cylinderprojection.

FIG. 16 is a flow diagram illustrating a process used in someimplementations of the present technology for object interaction with acone projection.

FIG. 17 is a conceptual diagram illustrating example cone projections.

FIG. 18 is a flow diagram illustrating a process used in someimplementations of the present technology for context-based selection ofa projection type.

FIG. 19 is a flow diagram illustrating a process used in someimplementations of the present technology for selecting objects using acut gesture.

FIG. 20 is a conceptual diagram illustrating an example of selectingobjects using a cut gesture.

FIG. 21 is a flow diagram illustrating a process used in someimplementations of the present technology for actuating objects using acut-click gesture.

FIG. 22 is a conceptual diagram illustrating an example of actuatingobjects using a cut-click gesture.

FIG. 23 is a flow diagram illustrating a process used in someimplementations of the present technology for selecting objects using alasso gesture.

FIGS. 24A and 24B are conceptual diagrams illustrating an example ofselecting objects using a lasso gesture.

FIG. 25 is a flow diagram illustrating a process used in someimplementations of the present technology for selecting objects using aframe gesture.

FIGS. 26A and 26B are conceptual diagrams illustrating an example ofselecting objects using a frame gesture.

FIG. 27 is a flow diagram illustrating a process used in someimplementations of the present technology for disambiguating betweentarget and non-target objects that intersect with a ray projection.

FIGS. 28 and 29 are conceptual diagrams illustrating an example hookdisambiguation between selected and non-selected objects that intersectwith a ray projection.

FIG. 30 is a flow diagram illustrating a process used in someimplementations of the present technology for disambiguating, with a“trombone” gesture, between target and non-target objects that intersecta cone or cylinder projection.

FIGS. 31 and 32 are conceptual diagrams illustrating an example trombonegesture to disambiguate between selected and non-selected objects thatintersect with a cone projection.

FIGS. 33 and 34 are conceptual diagrams illustrating a second exampletrombone gesture to disambiguate between selected and non-selectedobjects that intersect with a cylinder projection.

FIG. 35 is a flow diagram illustrating a process used in someimplementations of the present technology for a bimanual interactionwith an object using a menu attached to a dominant hand.

FIGS. 36, 37 and 38 are conceptual diagrams illustrating an example of abimanual interaction with an object using a menu attached to a dominanthand.

FIG. 39 is a flow diagram illustrating a process used in someimplementations of the present technology for switching input modalitiesfor global or local mode interactions.

FIGS. 40A and 40B are conceptual diagrams illustrating an example ofswitching input modalities for global or local mode interactions.

FIG. 41 is a flow diagram illustrating a process used in someimplementations of the present technology for interpreting gestures asinteractions in either global or local mode.

FIGS. 42A and 42B are conceptual diagrams illustrating an example ofinterpreting gestures as interactions in either global or local mode.

The techniques introduced here may be better understood by referring tothe following Detailed Description in conjunction with the accompanyingdrawings, in which like reference numerals indicate identical orfunctionally similar elements.

DETAILED DESCRIPTION

Embodiments for interacting with objects in an artificial realityenvironment are described herein. In an artificial reality environment,users may want to perform indirect interactions where the cause andeffect of the interaction are spatially separated. The presenttechnology provides projections that a user can create to perform objectinteractions. For example, when a user wishes to interact with an objectoutside her immediate reach, she can use a projection to select, move,or otherwise interact with the distant object. The present technologyalso includes object selection techniques for identifying anddisambiguating between target objects, allowing a user to select objectsboth near and distant from the user. Yet further aspects of the presenttechnology include techniques for interpreting various bimanual(two-handed) gestures for increased functionality and control wheninteracting with objects. The present technology further includes amodel for differentiating between global and local modes for, e.g.,providing different input modalities or interpretations of usergestures.

A projection is an indication in two dimensions (e.g., a line or “ray”)or an area in three dimensions (e.g., a cylinder, sphere, cone, pyramid,etc.) which can specify objects with which a user wishes to interact.For example, a hand interaction system can identify as target objectsone or more objects that intersect a projection. Interpreting hand andbody gestures to correctly interact with objects outside a user'simmediate reach can be difficult due to inaccuracies in tracking handand body positions and postures. These difficulties are compounded dueto the exponential effect that small hand movements have on a projectionposition as the projection extends outward from the user. To increaseprojection stability, the hand interaction system can control aprojection position based on an origin point and a control point. Thehand interaction system can set a line extending from the origin pointthrough the control point as the center of a projection. In variousimplementations, the origin point can be a tracked part of a user'sbody, such as a dominant eye, a hip, or a shoulder, and the controlpoint can be a part of a user's hand such as fingertips, a palm, awrist, or a fist. In some implementations, the origin point can also bea predicted location of a body part (as opposed to an expressly trackedbody part). This prediction can be acquired through the use of aninverse kinematic model. In some implementations, the origin point canchange based on context such as where the user is directing her gaze oran angle of the projection center line.

The hand interaction system can provide various projection types. A rayprojection can be a single line extending, e.g., from a user's hand intothe artificial reality environment. A ray projection can provide veryaccurate interactions but with low precision, particularly at adistance. A sphere projection can be a ray projection with a sphere atthe end of it and a cylinder projection can be a cylinder extending,e.g., from a user's hand, into the artificial reality environment. Acone or pyramid projection can be a cone or pyramid with the tipextending, e.g., from a user's hand or eye, and the projectionincreasing exponentially in coverage area as the projection extends intothe artificial reality environment. Due to their coverage of a largerarea than a ray projection, projections such as cylinders, spheres,cones, and pyramids can be easier to use to correctly indicate an objectwith which a user wishes to interact, but generally with less accuracy.

In some situations, a projection can intersect with more objects thanthose with which a user wishes to interact. For example, a user may wishto select only some of a tight cluster of objects. In other situations,a projection can fail to intersect objects with which a user wishes tointeract. For example, a projection can be configured not to passthrough objects and a user may wish to select an object occluded byanother object. The hand interaction system can include various bimanual(two-handed) techniques for disambiguating between selected objects orextending the coverage of a projection. In some implementations, thehand interaction system can specify a “hook” at a point on a rayprojection. The distance on the ray projection between the user'sdominant hand and the hook can be based on another gesture, such as thedistance between the user's dominant and non-dominant hand or a distancebetween two of the user's fingers. In other implementations, the handinteraction system can control the shape of a projection based onanother gesture. For example, the hand interaction system can change thediameter of a cylinder, sphere, or a base of a cone projection based onanother gesture, such as the distance between the user's dominant andnon-dominant hand or a distance between two of the user's fingers.

In various implementations, the hand interaction system can also detectadditional object interaction gestures such as a cut, a cut-click, alasso, or a frame gesture. In some implementations, these gestures canuse a ray projection configured with an origin point at the user'sdominant eye and a control point on her hand, so it appears to the useras if she is “drawing” into the environment. A cut gesture is where auser moves a projection to intersect with one or more objects. The handinteraction system can select each object that intersects with theprojection. A cut-click gesture is where a user moves a projection tointersect with one or more actionable controls (e.g., a button). Thehand interaction system can actuate each control that intersects withthe projection. A lasso gesture is where the hand interaction systemidentifies a lasso movement such as by comparing a user movement topreviously observed gestures (e.g., using a machine learning model) orby identifying that a user has made a closed shape gesture. The handinteraction system can identify a projection (e.g., a distorted cone)with a point extending from a user's eye and the distorted cone shapedefined by the lasso gesture. The hand interaction system can selectobjects that intersect with the distorted cone. A frame gesture is wherethe hand interaction system identifies a frame input such as byidentifying that a user has defined the edges of a rectangle with thethumb and index finger of both hands or by pulling apart oppositecorners of the rectangle. The hand interaction system can identify aprojection (e.g., a pyramid) with a point extending from a user's eyeand the pyramid walls defined by the rectangle. The hand interactionsystem can select objects that intersect with the pyramid.

The hand interaction system can also identify whether the user isattempting to interact with objects in various operational modes, e.g.,a global mode or a local mode. A global mode can be for manipulatingobjects within an environment, e.g., selecting objects, controllingobject placement in the virtual environment, etc. A local mode can befor interacting with interfaces of one or more selected objects, e.g.,pressing buttons on the objects, interacting with menus or othercontrols associated with the objects, etc. In various implementations,the user can explicitly select between global and local modes (e.g.,with a gesture or button press) or global/local mode selection can beimplicit (e.g., based on a user's gaze). In some implementations, aninput modality can be automatically changed based on which operationalmode is activated. For example, in global mode, inputs can be switchedto projection casting, such as ray, cylinder, sphere, or coneprojections. In local mode, inputs can be switched to other modalities,such as by mapping controls associated with selected objects to fingerson the user's hand, where a particular detected movement of the mappedfinger controls actuation of the corresponding control. In someimplementations, actions that can be taken with respect to an object canbe divided into being performable in a particular operational mode. Thehand interaction system can match a gesture to an action in the set ofactions defined for the currently activated mode. For example, when inglobal mode, a gesture dragging downward from an upper right corner of avirtual chess board can be interpreted as an intention to rotate thechess board. However, the same gesture, when in local mode, can beinterpreted as moving a rook from the upper right corner square forwardan amount corresponding to the length of the gesture.

Embodiments of the disclosed technology may include or be implemented inconjunction with an artificial reality system. Artificial reality orextra reality (XR) is a form of reality that has been adjusted in somemanner before presentation to a user, which may include, e.g., a virtualreality (VR), an augmented reality (AR), a mixed reality (MR), a hybridreality, or some combination and/or derivatives thereof. Artificialreality content may include completely generated content or generatedcontent combined with captured content (e.g., real-world photographs).The artificial reality content may include video, audio, hapticfeedback, or some combination thereof, any of which may be presented ina single channel or in multiple channels (such as stereo video thatproduces a three-dimensional effect to the viewer). Additionally, insome embodiments, artificial reality may be associated withapplications, products, accessories, services, or some combinationthereof, that are, e.g., used to create content in an artificial realityand/or used in (e.g., perform activities in) an artificial reality. Theartificial reality system that provides the artificial reality contentmay be implemented on various platforms, including a head-mounteddisplay (HMD) connected to a host computer system, a standalone HMD, amobile device or computing system, a “cave” environment or otherprojection system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

“Virtual reality” or “VR,” as used herein, refers to an immersiveexperience where a user's visual input is controlled by a computingsystem. “Augmented reality” or “AR” refers to systems where a user viewsimages of the real world after they have passed through a computingsystem. For example, a tablet with a camera on the back can captureimages of the real world and then display the images on the screen onthe opposite side of the tablet from the camera. The tablet can processand adjust or “augment” the images as they pass through the system, suchas by adding virtual objects. “Mixed reality” or “MR” refers to systemswhere light entering a user's eye is partially generated by a computingsystem and partially composes light reflected off objects in the realworld. For example, a MR headset could be shaped as a pair of glasseswith a pass-through display, which allows light from the real world topass through a waveguide that simultaneously emits light from aprojector in the MR headset, allowing the MR headset to present virtualobjects intermixed with the real objects the user can see. “Artificialreality,” “extra reality,” or “XR,” as used herein, refers to any of VR,AR, MR, or any combination or hybrid thereof.

There are existing XR systems for selecting and interacting with objectsin artificial reality environments. However, these XR systems areinaccurate, imprecise, and provide limited functionality. Existing XRsystems often require separate controller devices for input and oftenfail to correctly interpret user gestures in relation to objects in theartificial reality environment. The existing XR systems are frustratingand time-consuming for users to operate. The hand interaction system andprocesses described herein overcome these problems associated withconventional XR interaction techniques and are expected to provide userswith greater control over object interactions, offer more functionality,and are more natural and intuitive than interactions in existing XRsystems. Despite being natural and intuitive, the XR systems andprocesses described herein are rooted in computerized artificial realitysystems, instead of being an analog of traditional object interactions.For example, existing object interaction techniques fail to describeprojection positioning based on multiple body parts or gestures, muchless provide for users to resize and customize such projections.Furthermore, existing XR systems do not provide effective selection anddisambiguation techniques, nor do they provide options for switchinginput modalities or interpreting user input based on a user intentionmode (e.g., global and local modes).

Several implementations are discussed below in more detail in referenceto the figures. FIG. 1 is a block diagram illustrating an overview ofdevices on which some implementations of the disclosed technology canoperate. The devices can comprise hardware components of a computingsystem 100 that interpret hand gestures for interacting with objects inan artificial reality environment. In various implementations, computingsystem 100 can include a single computing device 103 or multiplecomputing devices (e.g., computing device 101, computing device 102, andcomputing device 103) that communicate over wired or wireless channelsto distribute processing and share input data. In some implementations,computing system 100 can include a stand-alone headset capable ofproviding a computer created or augmented experience for a user withoutthe need for external processing or sensors. In other implementations,computing system 100 can include multiple computing devices such as aheadset and a core processing component (such as a console, mobiledevice, or server system) where some processing operations are performedon the headset and others are offloaded to the core processingcomponent. Example headsets are described below in relation to FIGS. 2Aand 2B. In some implementations, position and environment data can begathered only by sensors incorporated in the headset device, while inother implementations one or more of the non-headset computing devicescan include sensor components that can track environment or positiondata.

Computing system 100 can include one or more processor(s) 110 (e.g.,central processing units (CPUs), graphical processing units (GPUs),holographic processing units (HPUs), etc.) Processors 110 can be asingle processing unit or multiple processing units in a device ordistributed across multiple devices (e.g., distributed across two ormore of computing devices 101-103).

Computing system 100 can include one or more input devices 120 thatprovide input to the processors 110, notifying them of actions. Theactions can be mediated by a hardware controller that interprets thesignals received from the input device and communicates the informationto the processors 110 using a communication protocol. Each input device120 can include, for example, a mouse, a keyboard, a touchscreen, atouchpad, a wearable input device (e.g., a haptics glove, a bracelet, aring, an earring, a necklace, a watch, etc.), a camera (or otherlight-based input device, e.g., an infrared sensor), a microphone, orother user input devices.

Processors 110 can be coupled to other hardware devices, for example,with the use of an internal or external bus, such as a PCI bus, SCSIbus, or wireless connection. The processors 110 can communicate with ahardware controller for devices, such as for a display 130. Display 130can be used to display text and graphics. In some implementations,display 130 includes the input device as part of the display, such aswhen the input device is a touchscreen or is equipped with an eyedirection monitoring system. In some implementations, the display isseparate from the input device. Examples of display devices are: an LCDdisplay screen, an LED display screen, a projected, holographic, oraugmented reality display (such as a heads-up display device or ahead-mounted device), and so on. Other I/O devices 140 can also becoupled to the processor, such as a network chip or card, video chip orcard, audio chip or card, USB, firewire or other external device,camera, printer, speakers, CD-ROM drive, DVD drive, disk drive, etc.

Computing system 100 can include a communication device capable ofcommunicating wirelessly or wire-based with other local computingdevices or a network node. The communication device can communicate withanother device or a server through a network using, for example, TCP/IPprotocols. Computing system 100 can utilize the communication device todistribute operations across multiple network devices.

The processors 110 can have access to a memory 150, which can becontained on one of the computing devices of computing system 100 or canbe distributed across of the multiple computing devices of computingsystem 100 or other external devices. A memory includes one or morehardware devices for volatile or non-volatile storage, and can includeboth read-only and writable memory. For example, a memory can includeone or more of random access memory (RAM), various caches, CPUregisters, read-only memory (ROM), and writable non-volatile memory,such as flash memory, hard drives, floppy disks, CDs, DVDs, magneticstorage devices, tape drives, and so forth. A memory is not apropagating signal divorced from underlying hardware; a memory is thusnon-transitory. Memory 150 can include program memory 160 that storesprograms and software, such as an operating system 162, hand interactionsystem 164, and other application programs 166. Memory 150 can alsoinclude data memory 170 that can include, e.g., projection descriptors,gesture identifiers (e.g., trained machine learning models) and/orgesture paths, mappings between gestures and actions, mappings betweenactions and user intention modes, configuration data, settings, useroptions or preferences, etc., which can be provided to the programmemory 160 or any element of the computing system 100.

Some implementations can be operational with numerous other computingsystem environments or configurations. Examples of computing systems,environments, and/or configurations that may be suitable for use withthe technology include, but are not limited to, XR headsets, personalcomputers, server computers, handheld or laptop devices, cellulartelephones, wearable electronics, gaming consoles, tablet devices,multiprocessor systems, microprocessor-based systems, set-top boxes,programmable consumer electronics, network PCs, minicomputers, mainframecomputers, distributed computing environments that include any of theabove systems or devices, or the like.

FIG. 2A is a wire diagram of a virtual reality head-mounted display(HMD) 200, in accordance with some embodiments. The HMD 200 includes afront rigid body 205 and a band 210. The front rigid body 205 includesone or more electronic display elements of an electronic display 245, aninertial motion unit (IMU) 215, one or more position sensors 220,locators 225, and one or more compute units 230. The position sensors220, the IMU 215, and compute units 230 may be internal to the HMD 200and may not be visible to the user. In various implementations, the IMU215, position sensors 220, and locators 225 can track movement andlocation of the HMD 200 in the real world and in a virtual environmentin three degrees of freedom (3DoF) or six degrees of freedom (6DoF). Forexample, the locators 225 can emit infrared light beams which createlight points on real objects around the HMD 200. One or more cameras(not shown) integrated with the HMD 200 can detect the light points.Compute units 230 in the HMD 200 can use the detected light points toextrapolate position and movement of the HMD 200 as well as to identifythe shape and position of the real objects surrounding the HMD 200.

The electronic display 245 can be integrated with the front rigid body205 and can provide image light to a user as dictated by the computeunits 230. In various embodiments, the electronic display 245 can be asingle electronic display or multiple electronic displays (e.g., adisplay for each user eye). Examples of the electronic display 245include: a liquid crystal display (LCD), an organic light-emitting diode(OLED) display, an active-matrix organic light-emitting diode display(AMOLED), a display including one or more quantum dot light-emittingdiode (QOLED) sub-pixels, a projector unit (e.g., microLED, LASER,etc.), some other display, or some combination thereof.

In some implementations, the HMD 200 can be coupled to a core processingcomponent such as a personal computer (PC) (not shown) and/or one ormore external sensors (not shown). The external sensors can monitor theHMD 200 (e.g., via light emitted from the HMD 200) which the PC can use,in combination with output from the IMU 215 and position sensors 220, todetermine the location and movement of the HMD 200.

In some implementations, the HMD 200 can be in communication with one ormore other external devices, such as controllers (not shown) which auser can hold in one or both hands. The controllers can have their ownIMU units, position sensors, and/or can emit further light points. TheHMD 200 or external sensors can track these controller light points. Thecompute units 230 in the HMD 200 or the core processing component canuse this tracking, in combination with IMU and position output, tomonitor hand positions and motions of the user. The controllers can alsoinclude various buttons a user can actuate to provide input and interactwith virtual objects. In various implementations, the HMD 200 can alsoinclude additional subsystems, such as an eye tracking unit, an audiosystem, various network components, etc. In some implementations,instead of or in addition to controllers, one or more cameras includedin the HMD 200 or external to it can monitor the positions and poses ofthe user's hands to determine gestures and other hand and body motions.

FIG. 2B is a wire diagram of a mixed reality HMD system 250 whichincludes a mixed reality HMD 252 and a core processing component 254.The mixed reality HMD 252 and the core processing component 254 cancommunicate via a wireless connection (e.g., a 60 GHz link) as indicatedby link 256. In other implementations, the mixed reality system 250includes a headset only, without an external compute device or includesother wired or wireless connections between the mixed reality HMD 252and the core processing component 254. The mixed reality HMD 252includes a pass-through display 258 and a frame 260. The frame 260 canhouse various electronic components (not shown) such as light projectors(e.g., LASERs, LEDs, etc.), cameras, eye-tracking sensors, MEMScomponents, networking components, etc.

The projectors can be coupled to the pass-through display 258, e.g., viaoptical elements, to display media to a user. The optical elements caninclude one or more waveguide assemblies, reflectors, lenses, mirrors,collimators, gratings, etc., for directing light from the projectors toa user's eye. Image data can be transmitted from the core processingcomponent 254 via link 256 to HMD 252. Controllers in the HMD 252 canconvert the image data into light pulses from the projectors, which canbe transmitted via the optical elements as output light to the user'seye. The output light can mix with light that passes through the display258, allowing the output light to present virtual objects that appear asif they exist in the real world.

Similarly to the HMD 200, the HMD system 250 can also include motion andposition tracking units, cameras, light sources, etc., which allow theHMD system 250 to, e.g., track itself in 3DoF or 6DoF, track portions ofthe user (e.g., hands, feet, head, or other body parts), map virtualobjects to appear as stationary as the HMD 252 moves, and have virtualobjects react to gestures and other real-world objects.

FIG. 3 is a block diagram illustrating an overview of an environment 300in which some implementations of the disclosed technology can operate.Environment 300 can include one or more client computing devices 305A-D,examples of which can include computing system 100. In someimplementations, some of the client computing devices (e.g., clientcomputing device 305B) can be the HMD 200 or the HMD system 250. Clientcomputing devices 305 can operate in a networked environment usinglogical connections through network 330 to one or more remote computers,such as a server computing device.

In some implementations, server 310 can be an edge server which receivesclient requests and coordinates fulfillment of those requests throughother servers, such as servers 320A-C. Server computing devices 310 and320 can comprise computing systems, such as computing system 100. Thougheach server computing device 310 and 320 is displayed logically as asingle server, server computing devices can each be a distributedcomputing environment encompassing multiple computing devices located atthe same or at geographically disparate physical locations.

Client computing devices 305 and server computing devices 310 and 320can each act as a server or client to other server/client device(s).Server 310 can connect to a database 315. Servers 320A-C can eachconnect to a corresponding database 325A-C. As discussed above, eachserver 310 or 320 can correspond to a group of servers, and each ofthese servers can share a database or can have their own database.Though databases 315 and 325 are displayed logically as single units,databases 315 and 325 can each be a distributed computing environmentencompassing multiple computing devices, can be located within theircorresponding server, or can be located at the same or at geographicallydisparate physical locations.

Network 330 can be a local area network (LAN), a wide area network(WAN), a mesh network, a hybrid network, or other wired or wirelessnetworks. Network 330 may be the Internet or some other public orprivate network. Client computing devices 305 can be connected tonetwork 330 through a network interface, such as by wired or wirelesscommunication. While the connections between server 310 and servers 320are shown as separate connections, these connections can be any kind oflocal, wide area, wired, or wireless network, including network 330 or aseparate public or private network.

FIG. 4 is a block diagram illustrating components 400 which, in someimplementations, can be used in a system employing the disclosedtechnology. Components 400 can be included in one device of computingsystem 100 or can be distributed across multiple of the devices ofcomputing system 100. The components 400 include hardware 410, mediator420, and specialized components 430. As discussed above, a systemimplementing the disclosed technology can use various hardware includingprocessing units 412, working memory 414, input and output devices 416(e.g., cameras, displays, IMU units, network connections, etc.), andstorage memory 418. In various implementations, storage memory 418 canbe one or more of: local devices, interfaces to remote storage devices,or combinations thereof. For example, storage memory 418 can be one ormore hard drives or flash drives accessible through a system bus or canbe a cloud storage provider (such as in storage 315 or 325) or othernetwork storage accessible via one or more communications networks. Invarious implementations, components 400 can be implemented in a clientcomputing device such as client computing devices 305 or on a servercomputing device, such as server computing device 310 or 320.

Mediator 420 can include components which mediate resources betweenhardware 410 and specialized components 430. For example, mediator 420can include an operating system, services, drivers, a basic input outputsystem (BIOS), controller circuits, or other hardware or softwaresystems.

Specialized components 430 can include software and/or hardwareconfigured to perform operations for recognizing gestures andinterpreting them to interact with objects in an artificial realityenvironment. Specialized components 430 can include, for example,projection casting engine 434, object selection engine 436, two-handedinteraction engine 438, global and local mode engine 440, and componentsand APIs which can be used for providing user interfaces, transferringdata, and controlling the specialized components, such as interface 432.In some implementations, components 400 can be in a computing systemthat is distributed across multiple computing devices or can be aninterface to a server-based application executing one or more ofspecialized components 430.

Projection casting engine 434 can create various types of projectionssuch as ray projections (as described in more detail below in relationto FIGS. 12 and 13), sphere or cylinder projections (as described inmore detail below in relation to FIGS. 14 and 15), or cone projections(as described in more detail below in relation to FIGS. 16 and 17). Insome implementations, projection casting engine 434 can select aprojection type based on a current context (as described in more detailbelow in relation to FIG. 18).

Projection casting engine 434 can also control projection positionsbased on multiple tracked body parts, such as part of a hand as acontrol point and another point on a user's body, such as an eye,shoulder, or hip, as an origin point (as described in more detail belowin relation to FIGS. 5-9). For example, the projection casting engine434 can set a beginning point for a projection at, or offset from, acontrol point and can determine a line connecting the origin point andthe control point as the center line for the projection. In someimplementations, the projection casting engine 434 can move the originfor a projection according to origin transition factors such as theuser's gaze direction, an angle of the projection, or a position oftarget objects (as described in more detail below in relation to FIGS.10 and 11).

Object selection engine 436 can identify interactions that selectmultiple objects. In some implementations, object selection engine 436can identify a cut gesture where a user draws a line that appears to theuser as intersecting with multiple objects (as described in more detailbelow in relation to FIGS. 19 and 20). In some implementations, objectselection engine 436 can identify a cut-click gesture where a user drawsa line that appears to the user as intersecting with multiple objects,and the system actuates those objects (as described in more detail belowin relation to FIGS. 21 and 22). In some implementations, objectselection engine 436 can identify a lasso gesture that forms, from theuser's perspective, a shape around multiple objects which the systemselects (as described in more detail below in relation to FIGS. 23 and24). In some implementations, object selection engine 436 can identify aframe gesture from a user specifying two opposite corners of therectangle, and the system can then select objects that appear to bewithin the rectangle from the user's point of view (as described in moredetail below in relation to FIGS. 25 and 26).

Two-handed interaction engine 438 can identify gestures performed by twohands (or different parts of the same hand) or gestures where a secondhand (or different part of a first hand) modifies results of a gestureperformed by a first hand (or a first part of the first hand). In someimplementations, two-handed interaction engine 438 can identify a hookgesture in which a point on a ray projection is identified as a hook.The hook can define which objects the ray selects or otherwise interactswith. The position of the hook along the ray can be specified based onthe distance between the two user's hands or the distance between twofingers on the same hand. Hook gestures are described in more detailbelow in relation to FIGS. 27-29. In some implementations, two-handedinteraction engine 438 can identify a trombone gesture in which adiameter of a cone, cylinder, or sphere projection is based on thedistance between the two user's hands or the distance between twofingers on the same hand. Trombone gestures are described in more detailbelow in relation to FIGS. 30-34. In some implementations, two-handedinteraction engine 438 can facilitate a bimanual interaction where auser selects an object using a gesture with a first hand, a menu ispresented in association with the first hand for interacting with theselected object, and the user can interact with the object with hersecond hand. Menus attached to a selection hand are described in moredetail below in relation to FIGS. 35-38.

Global and local mode engine 440 can identify whether the handinteraction system is in a global or local mode, e.g., based on anexplicit user command or a context, such as how long a user has had aconsistent gaze directed toward an object. In some implementations,global and local mode engine 440 can switch input modalities for globaland local modes (as described in more detail below in relation to FIGS.39 and 40). In some implementations, global and local mode engine 440can interpret ambiguous input based on whether the hand interactionsystem is in global or local mode (as described in more detail below inrelation to FIGS. 41 and 42).

Those skilled in the art will appreciate that the components illustratedin FIGS. 1-4 described above, and in each of the flow diagrams discussedbelow, may be altered in a variety of ways. For example, the order ofthe logic may be rearranged, substeps may be performed in parallel,illustrated logic may be omitted, other logic may be included, etc. Insome implementations, one or more of the components described above canexecute one or more of the processes described below.

Projection Casting with Origin and Control Point

When using a controller to point at and select objects in an artificialreality environment, it is easy to set and track a single point on thecontroller from which a straight line can be defined. Existingcontroller-based selection techniques are not directly transferrable tobare hands, as there are inherent differences between how peopleapproach using a tool versus theft own body. When a user's hands are theinput method, it is much more difficult to track a single point as aprojection source. Furthermore, unlike a rigid controller surface, auser's hands tend to make many small movements which can cause aprojection originating from the user's hand to jump around and thesemovements are amplified as the projection extends further from the user.These natural hand tremors and tracking issues can interfere withprojection casting and cause inaccuracies in object selection.

To increase accuracy of projection casting based on hand tracking,projection positions can be defined using both a primary control point(e.g., part of a user's hand) and a secondary origin point that isoutside of the user's bare hands to anchor the projection. Projectionsbased on a combination of these two positions results in a more stableand precise casting experience. Various origin points can be used andcan provide different typical input speed, precision, and ergonomicsdepending on the size, distance, vertical position and density of thetarget objects. In various implementations, origin points can include auser's dominant eye, a point between the user's eyes, another point onthe user's head, the user's hip, the user's shoulder, or a contextvariable point, e.g., between the user's hip and shoulder. Each of thevarious origin points can be paired with various control points such asthe tips of a user's finger when making a particular gesture, a user'spalm, a user's wrist, or a center of a user's fist. When casting aprojection, the projection can be cast outward from the user, startingfrom the control point and oriented to be centered on a line that passesthrough both the origin point and the control point.

FIG. 5 is a flow diagram illustrating a process 500 used in someimplementations of the present technology for casting a projection withan origin point and a control point. At block 502, process 500 can tracka portion of a hand as a control point. For example, a control point canbe the tips of a user's finger, a user's palm, a user's wrist, or acenter of a user's fist. In some implementations, the control point canbe identified in response to a user making a particular gesture, such asforming her fingers into a “pinch.” In some cases, the control point canbe offset from a tracked portion of the user. For example, the controlpoint can be an offset from the user's palm or wrist. This helpseliminate jitter that would occur in instances that use finger tips,i.e. when the control point would move when forming the gesture.Instead, a point offset six inches forward of the wrist moves less whenforming such a gesture with the fingers. In some implementations, acontrol point can be based on other body parts, such as a point on theuser's head or a portion of a foot.

At block 504, process 500 can track a second body part as an originpoint. In various implementations, and origin point can be based on aposition of a user's eye, shoulder, hip, etc. In some implementations,the eye, shoulder or hip used can be the one on the user's dominantside. The user's dominant eye, for example, can be determined based onwhich hand a user tends to employ the most, a user stating which hand oreye is dominant (e.g., in an onboarding procedure), observances of howthe user has positioned herself in relation to past stimulus, visualtests provided to the user, etc. In some implementations, the originpoint can move depending on circumstances such as where the user's gazeis directed, an angle of a projection user is creating, or a selectionfrom a machine learning model. Adjusting an origin point is discussed ingreater detail below in relation to FIGS. 10 and 11.

The origin point and control point can be tracked, at block 502 and 504,by analyzing images from a camera, e.g., integrated with an XR headsetor in an external sensor. For example, a machine learning model can beused to analyze images from such a camera and to generate 3D positiondata for a model of the user's hands or other various body parts. Insome implementations, additional or alternative methods can be applied,such as by using position or IMU tracking sensors in the headset, in abracelet, in a glove, or other wearable device.

At block 506, process 500 can determine a projection orientation that iscentered on a line the passes through A) the origin point determined atblock 502 and B) the control point determined at block 504. As discussedbelow, the projection can be one of various types such as a ray, sphere,cylinder, cone, pyramid, etc. The projection can extend outward from theuser starting at the control point or offset from the control point.

At block 508, process 500 can perform an action in relation to real orvirtual objects, based on one or more locations of the projection. Forexample, actions can include any interaction such as selection ofobjects, moving or rotating objects, interacting with controls (e.g.,pressing buttons, pulling or pushing controls), etc. In some cases, theaction can be based on multiple locations of the projection, e.g., asthe user moves the control point and/or origin point, causingcorresponding movement of the projection. For example, as a user movesher palm, which is tracked as the control point with her shoulder as thecorresponding origin point, a ray projection cast outward from theuser's palm can intersect with multiple objects, each of which can beselected.

FIG. 6 is a conceptual diagram illustrating an example 600 of a rayprojection 606 with a shoulder origin point 602 and palm control point606. FIG. 7 is a conceptual diagram illustrating an example 700 of a rayprojection 706 with a dominant eye origin point 702 and a fingertipcontrol point 704. FIG. 8 is a conceptual diagram illustrating anexample 800 of a ray projection 806 with a hip origin point 802 and afist control point 804. FIG. 9 is a conceptual diagram illustrating anexample 900 of a ray projection 906 with a shoulder origin point 802 anda wrist control point 804. In each example 600, 700, 800, or 900, as theuser moves her tracked origin point (shoulder, eye, or hip) and/ortracked control point (wrist, palm, or fingertips), correspondingmovement to projection 606, 706, 806, or 906 will occur.

Different origin points can work better in different contexts. Forexample, higher origin points afford easier access to lower targets,while lower origin points enable easier access towards higher targets.In some implementations, the hand interaction system can identifyconditions and move the origin point accordingly. In variousimplementations, the origin point can snap between different originpoints or can move at a given speed on a path between origin points.

FIG. 10 is a flow diagram illustrating a process 1000 used in someimplementations of the present technology for adjusting a projectionorigin point. At block 1002, process 1000 can identify a gaze directionand/or other origin transition factors. In various implementations, theorigin transition factors can be one or more of: a user's gaze,horizontal angle of a current projection, location of target objects inrelation to the current origin point; etc. For example, process 1000 candetermine an angle of a centerline of a current projection. As anotherexample, process 1000 can determine that a current projection selectsobjects and that those objects are above or below a threshold plane. Insome implementations, transition factors can be adjusted based on userbiometrics (e.g. height, torso length, arm length, etc.) provided eitherthrough manual entry, calibration, or automated recognition. Inaddition, in some cases, continuous tracking can be used to moreaccurately determine whether, for example, forearms are outstretched, orrelaxed, and their specific angle (rather than using an average armlength and predicting this from the tracked hand position).

At block 1004, process 1000 can compare origin transition factorsdetermined at block 1002 to thresholds to determine if the origin pointshould be set higher or lower. For example, when the user's gaze isabove a threshold angle from their body, a high interaction mode can beselected and when the user's gaze is below the threshold angle fromtheir body, a low interaction mode can be selected. As another example,process 1000 can determine that an angle of a centerline of a currentprojection is greater than a threshold angle (e.g., pointing upward) andcan select a high interaction mode or can determine that the angle ofthe centerline of the current projection is less than a threshold angle(e.g., pointing downward) and can select a low interaction mode. As yetanother example, process 1000 can determine that a current projectionselects objects that are above a threshold plane and can select a highinteraction mode or can determine that the current projection selectsobjects that are below a threshold plane and can select a lowinteraction mode.

At block 1006, process 1000 can select a high interaction mode where theorigin point is transitioned toward a low location such as the user'ship. In some implementations, high interaction mode can makeinteractions with objects above a certain plane easier. In someimplementations, the transition can be a snap of the origin point to thelow location. In some implementations, there is a path specified betweentwo points (a high and a low point) on the user's body, such as theuser's shoulder and hip. Process 1000 can at block 1006 begin moving theorigin point at a set pace toward the low end of the path, unless originpoint is already at the lowest spot on the path.

At block 1008, process 1000 can select a low interaction mode where theorigin point is transitioned toward a high location, such as the user'seye or shoulder. In some implementations, low interaction mode can makeinteractions with objects below a certain plane easier. In someimplementations, the transition can be a snap of the origin point to thehigh location. In some implementations, there is a path specifiedbetween two points (a high and a low point) on the user's body, such asthe user's eye and hip. Process 1000 can, at block 1008, begin movingthe origin point at a set pace toward the high end of the path, unlessthe origin point is already at the highest spot on the path.

Process 1000 can repeat, returning to block 1002, as new origintransition factors are determined.

FIGS. 11A, 11B and 11C are conceptual diagrams illustrating an exampleof adjusting an origin point 1004 for a ray projection that is alsobeing controlled by a control point 1102, where the origin point 1104moves along on a path 1106 between a hip and a shoulder. In example1100, the movement of the origin point 1104 is based on the user's eyegaze 1108. The point along path 1106 where the origin is set is based ona linear relationship of a proportion of the path with the angle formedby the user's eye gaze and a plane 1110, which is perpendicular with thefloor.

In FIG. 11A, the user's eye gaze 1108 is perpendicular with the plane1110 and the origin point 1104 is set so the ray between origin point1104 and control point 1102 is perpendicular to path 1106. In FIG. 11B,the user's eye gaze 1108 has traveled upward beyond a maximum upwardangle, so the origin point 1104 is set at the lowest point on path 1106.In FIG. 11C, the user's eye gaze 1108 has traveled downward below aminimum downward angle, so the origin point 1104 is set at the highestpoint on path 1106. In example 1100, the transitions between thesepoints occurs at a set speed, moving the origin point 1104 smoothlyalong path 1106.

Virtual Object Interaction at a Distance with Projection Types

There are a number of difficulties with existing XR systems wheninteracting with objects, particularly when those objects are outsidethe immediate reach of the user. For example, many existing handtracking systems are imprecise, causing virtual models of the user'shand to be either jittery or to lag behind the actual position of theuser's hand. In addition, users' hands tend to exhibit tremors and othersmall movements, which translate into big projection position changeswhen the projection is cast over a large distance. Yet anotherdifficulty, known as the Heisenberg effect, is the movement that oftenoccurs in a pointing device (hand, mouse, stylus, etc.) when a userperforms an input with the pointing device (makes a gesture, clicks abutton, etc.) In many instances involving interactions at a distance,these difficulties can be overcome by using a suitable projectioncasting type. A non-exhaustive list of casting types includes a rayprojection, a sphere projection, a cylinder projection, a coneprojection, and a pyramid projection.

A ray projection, for purposes of this disclosure, is a single lineextending from a control point or an offset from the control point. Aray projection has little ambiguity as to a user's target object.However, the small target size can make interactions fragile—users canfind it difficult to point at individual objects, particularly smallones that are outside the user's reach.

A sphere projection, for purposes of this disclosure, is a rayprojection with a sphere attached to the end. In some implementations, auser can control the length of the ray and/or the diameter of thesphere. In other implementations, the diameter of the sphere and/orlength of the ray can be fixed. In yet other implementations the lengthof the ray can be set so that the sphere intersects with the object(s)nearest the user along the ray. A cylinder projection is similar to asphere projection, except that the sphere is translated along the entirelength of the ray, forming a cylinder extending from the projectioncontrol point. Sphere and cylinder projections tend to improve selectionaccuracy, as compared to ray projections, for interactions with objectsjust outside the user's reach (e.g., 1-3 feet).

A cone projection, for purposes of this disclosure, is a cone extendingfrom the projection control point and growing larger in diameter as thecone extends away from the user. In some implementations, the degree towhich the diameter changes as the cone projection extends away from theuser can be fixed. In other implementations, a user can control thediameter of the cone, as discussed below in relation to FIGS. 30-34. Apyramid projection can be similar to a cone projection but can use arectangular base instead of a circular one.

In some cases when at least part of a projection is three-dimensionalsuch as a sphere, cylinder, cone, or pyramid, the target object(s) ofthe projection can be objects that the projection partially intersects,objects that the projection fully encompasses, or objects that theprojection encompasses by at least a threshold amount (e.g., 30%, 50%,or 80%). In some implementations where a projection target can bemultiple objects, secondary selection techniques can be used todifferentiate between which objects the user intends to target. Forexample, individual objects that intersect with a projection can bemapped to fingers on a user's hand, and movement of the mapped fingers(e.g., tapping finger to thumb) can cause the corresponding mappedobject to be added or removed from a set of target objects. As anotherexample, another input modality such as a user's gaze can differentiatewhich intersected objects become target objects. In someimplementations, gestures can be mapped to selection addition orselection removal—e.g., one hand gesture can be used to add to the setof selected objects, and another can be used (with the castingtechniques) to remove objects from the set of selected objects.

FIG. 12 is a flow diagram illustrating a process 1200 used in someimplementations of the present technology for object interaction using aray projection. At block 1202, process 1200 can determine a controlpoint and a casting direction, for a ray projection, based on a trackedposition of one or more body parts. In some implementations, asdiscussed above in relation to FIG. 5 through FIG. 11, the control pointcan be a palm, fingertips, a fist, a wrist, etc., and the castingdirection can be along a line that passes through the control point andan origin point, such as a shoulder, eye, or hip. In otherimplementations, the control point can be based on other tracked bodyparts such as a user's eye, head, or chest. For example, the controlpoint can be an estimated position of a center of a user's pupil and theorigin point can be an estimated position of the center of a user'sretina. In some cases, instead of using an origin point to define adirection, the system can define a plane corresponding to the controlpoint and can determine the casting direction that is perpendicular tothat plane. For example, where the control point is on the head ortorso, the system can identify the coronal plane corresponding to thehead or torso and can identify the casting direction as perpendicular tothat coronal plane.

At block 1204, process 1200 can generate a ray projection (i.e. straightline) from the control point along the casting direction. In some cases,a graphical representation of the ray projection (the whole line or justa point where the ray hits an object) can be displayed in the artificialreality environment, while in other cases the ray projection is trackedby the XR system without displaying the ray projection. In variousimplementations, the ray projection can extend from the control pointuntil it intersects with a first (real or virtual) object or the rayprojection can extend through multiple objects. In some implementations,the direction of the ray projection can be adjusted to “snap” to objectsthat it is close to intersecting or the ray projection can be curved upto a threshold amount to maintain intersection with such objects.

At block 1206, process 1200 can determine an intersection between theray projection and one or more target objects. In some implementations,process 1200 can determine the intersection when a user makes aparticular gesture, such as tapping an index finger to a thumb. In someimplementations, the target object can be the first object that the rayprojection intersects, excluding other objects occluded by the firstobject. In other implementations, the ray can extend through multipleobjects and the target objects can be all of the objects that the rayprojection intersects. In yet other implementations, objects that theray intersects can be identified as target objects depending on otherbody part positions or gestures, such as the user's gaze or gesturesinvolving fingers mapped to intersected objects.

At block 1208, process 1200 can cause an interaction with the one ormore target objects determined at block 1206. For example, process 1200can select the target objects, cause them to move or rotate, or performother interactions.

FIG. 13 is a conceptual diagram illustrating an example 1300 of a rayprojection 1302. In example 1300, a user's hand 1306 has formed agesture by connecting her thumb and middle finger, indicating initiationof a projection. In example 1300 the control point 1304 is an offsetfrom the user's middle finger. Ray projection 1302 extends from thecontrol point 1304 to the target object 1308. In example 1300, the userhas closed her index finger to her thumb (not shown) and reopened it,making a “click” gesture, causing target object 1308 to be selected.

FIG. 14 is a flow diagram illustrating a process 1400 used in someimplementations of the present technology for object interaction using acylinder projection. At block 1402, process 1400 can determine a controlpoint and a casting direction, for the cylinder projection, based on atracked position of one or more body parts. In some implementations, asdiscussed above in relation to FIG. 5 through FIG. 11, the control pointcan be a palm, fingertips, a fist, a wrist, etc. and the castingdirection can be along a line that passes through the control point andan origin point, such as a shoulder, eye, or hip. In otherimplementations, the control point can be based on other tracked bodyparts such as a user's eye, head, or chest. For example, the controlpoint can be an estimated position of a of a center of user's pupil andthe origin point can be an estimated position of the center of a user'sretina. In some cases, instead of using an origin point to define adirection, the system can define a plane corresponding to the controlpoint and can determine the casting direction as perpendicular to thatplane. For example, where the control point is on the head or torso, thesystem can identify the coronal plane corresponding to the head or torsoand can identify the casting direction as perpendicular to that coronalplane.

At block 1404, process 1400 can generate a cylinder projection from thecontrol point along the casting direction. In some implementations,instead of a cylinder, the projection can be a sphere projection, whichincludes a sphere at the end of a ray projection. In some cases, agraphical representation of the cylinder (or sphere) projection can bedisplayed in the artificial reality environment (e.g., showing theentire projection or where it intersects with objects), while in othercases the cylinder (or sphere) projection is tracked by the XR systemwithout displaying the projection. In various implementations, thecylinder projection can extend from the control point until itintersects with a first (real or virtual) object or the cylinderprojection can extend to at least partially intersect with multipleobjects. In some implementations, the ray of the sphere projection canextend from the control point until the sphere intersects with a first(real or virtual) object or the ray of the sphere projection can extenda fixed distance.

At block 1406, process 1400 can determine an intersection between thecylinder (or sphere) projection and one or more target objects. In someimplementations, process 1400 can determine the intersection when a usermakes a particular gesture, such as tapping an index finger to a thumb.In some implementations, the target object can be the first object thatthe cylinder (or sphere) projection at least partially intersects orwholly encompasses. In other implementations, the target objects can beall of the objects that the cylinder (or sphere) projection at leastpartially intersects or wholly encompasses. In yet otherimplementations, objects that the cylinder (or sphere) intersects can beidentified as target objects depending on other body part positions orgestures, such as the user's gaze or gestures involving fingers mappedto intersected objects. In some implementations, as discussed below inrelation to FIGS. 30-34, the diameter of the cylinder (or sphere) can becontrolled by other gestures, such as a distance between the user'shands or the distance between the tip of the user's index finger andthumb.

At block 1408, process 1400 can cause an interaction with the one ormore target objects determined at block 1406. For example, process 1400can select the target objects, cause them to move or rotate, or performother interactions.

FIG. 15 is a conceptual diagram illustrating an example 1500 of acylinder projection 1502. In example 1500, a user's hand 1506 has formeda gesture by connecting her thumb and middle finger, indicatinginitiation of a projection. In example 1500, the control point 1504 isan offset from the user's middle finger. Cylinder projection 1502extends from the control point 1504 to the target objects 1508,intersecting at least partially with each of target objects 1508A,15088, 1508C, 1508D, and 1508E. In example 1500, the user has made agesture by rotating her hand 1506 clockwise (not shown), causing targetobjects 1508A-E to be selected.

FIG. 16 is a flow diagram illustrating a process 1600 used in someimplementations of the present technology for object interaction using acone projection. At block 1602, process 1600 can determine a controlpoint and a casting direction, for a projection, based on a trackedposition of one or more body parts. In some implementations, asdiscussed above in relation to FIG. 5 through FIG. 11, the control pointcan be a palm, fingertips, a fist, a wrist, etc. and the castingdirection can be along a line that passes through the control point andan origin point, such as a shoulder, eye, or hip. In otherimplementations, the control point can be based on other tracked bodyparts such as a user's eye, head, or chest. For example, the controlpoint can be an estimated position of a of a center of user's pupil andthe origin point can be an estimated position of the center of a user'sretina. In some cases, instead of using an origin point to define adirection, the system can define a plane corresponding to the controlpoint and can determine the casting direction as perpendicular to thatplane. For example, where the control point is on the head or torso, thesystem can identify the coronal plane corresponding to the head or torsoand can identify the casting direction as perpendicular to that coronalplane.

At block 1604, process 1600 can generate a cone projection with a conetip at the control point and the cone extending away from the user alongthe casting direction. In some implementations, instead of a cone, theprojection can be a pyramid projection with a pyramid tip at the controlpoint and the pyramid extending along the casting direction. In somecases, a graphical representation of the cone (or pyramid) projectioncan be displayed in the artificial reality environment (e.g., arepresentation of the entire cone or pyramid or a representation ofwhere the projection intersects objects), while in other cases the cone(or pyramid) projection is tracked by the XR system without displayingthe projection. In various implementations, the cone (or pyramid)projection can extend from the control point until it intersects a first(real or virtual) object or the cone (or pyramid) projection can extendto at least partially intersect multiple objects.

At block 1606, process 1600 can determine an intersection between thecone (or pyramid) projection and one or more target objects. In someimplementations, process 1600 can determine the intersection when a usermakes a particular gesture, such as tapping an index finger to a thumb.In some implementations, the target object can be the first object thatthe cone (or pyramid) projection at least partially intersects or whollyencompasses. In other implementations, the target objects can be all ofthe objects that the cone (or pyramid) projection at least partiallyintersects or wholly encompasses. In yet other implementations, objectsthat the cone (or pyramid) intersects can be identified as targetobjects depending on other body part positions or gestures, such as theuser's gaze or gestures involving fingers mapped to intersected objects.In some implementations, as discussed below in relation to FIGS. 30-34,the diameter of the cone can be controlled by other gestures, such as adistance between the user's hands or the distance between the tip of theuser's index finger and thumb.

At block 1608, process 1600 can cause an interaction with the one ormore target objects determined at block 1606. For example, process 1600can select the target objects, cause them to move or rotate, or performother interactions.

FIG. 17 is a conceptual diagram illustrating an example 1700 of two coneprojections 1702 and 1752. A users' hands 1706 and 1756 have formedgestures by connecting a thumb and middle finger, indicating initiationof a projection. In example 1700 the control points 1704 and 1754 are anoffset from the users' middle fingers. Cone projection 1702 extends fromthe control point 1704, increasing in diameter until it intersects withtarget objects 1708, intersecting at least partially with each of targetobjects 1708A, 17088, 1708C, 1708D, and 1708E. In example 1700, objectsare selected as soon as the projection intersects with them, targetobjects 1708A-E are therefore selected by cone projection 1702. In someimplementations, a projection can only select the object that mostintersects the projection. In example 1700, this implementation wouldcause object 1708B to be selected. Cone projection 1752 extends from thecontrol point 1754, also increasing in diameter. However, target object1758 is closer to hand 1756 than target objects 1708 are to hand 1706,so cone projection 1752 only intersects with target object 1758. Asobjects are selected in example 1700 as soon as the projectionintersects with them, target object 1758 is selected by cone projection1752.

In some implementations, the hand interaction system can automaticallyselect a type of projection to cast based on a current context. Forexample, the system can use ray projections when apparent target objectsare closely grouped (e.g., when another type of projection wouldencompass greater than a threshold number of objects, or objects withina threshold distance from a center of a projection are within athreshold distance from each other). As another example, the handinteraction system can switch to sphere or cone projections when theclosest objects that a ray projection would intersect are greater than athreshold distance away. FIG. 18 is a flow diagram illustrating aprocess 1800 used in some implementations of the present technology forcontext-based selection of a projection type.

At block 1802, process 1800 can identify an interaction plane for a usergaze. An interaction plane is a plane in an artificial realityenvironment of an object (or for the center between multiple objects)with which a user is currently interacting. For example, an interactionplane can be determined based on a plane at which a tracked gaze of theuser is directed. As another example, if a user has selected a virtualobject, the interaction plane can be centered on that object. At block1804, process 1800 can compute a distance, in virtual space of theartificial reality environment, between the user and the interactionplane identified at block 1802.

At block 1806, process 1800 can identify a surface type of a currenttarget object. In various implementations, surface types can beclassified based on whether the surface of the target object is flat ornot, the size of the target object, whether the target object includescontrols, what type of controls the target object includes, what type ofinteractions the user or other users have previously performed inrelation to the target object, etc. In other implementations, a type canbe a setting assigned to the target object, e.g., by the creator of theobject. In some implementations, process 1800 can begin at block 1806instead of block 1802. In various implementations, process 1800 caninclude or skip block 1806.

At block 1808, process 1800 can select a projection type based on thedistance computed at block 1804 and/or the surface type identified atblock 1806. In some implementations, this selection can be based on amapping of distance ranges to projection types. For example, when theinteraction plane is within three feet of the user, ray projections canbe used; when the interaction plane is between three and seven feet ofthe user, cylinder projections can be used; and when the interactionplane is more than seven feet from the user, cone projections can beused. In other implementations, a mapping can be of surface types toprojection types. For example, where the surface type indicates a flatsurface with multiple buttons and controls, ray projections can be used,and otherwise cylinder projections can be used. In yet furtherimplementations, the mapping can be for pairs of distance ranges andsurface types to projection types. For example, when the interactionplane is within three feet of the user and the surface type indicates aflat surface with multiple buttons and controls, ray projections can beused; when the interaction plane is within three feet of the user andthe surface type indicates no controls, cylinder projections can beused; when the interaction plane is between three and seven feet fromthe user and the surface type indicates the object is larger than aseven-inch threshold cylinder projections can be used but if the surfacetype indicates the object is smaller than the seven-inch threshold rayprojections can be used; and if the interaction plane is further thanseven feet from the user, cone projections can be used without regard tothe surface type.

Multi-Object Selection

When a user operates in an artificial reality environment, she canselect various objects for further interaction or specify to the XRsystem to perform operations on objects. While some existing XR systemshave techniques to select objects, such as by performing a fingergesture directed at each object, these techniques have a number ofdrawbacks. For example, when a user wishes to select a group of objects,selecting each object individually is time-consuming, frustrating, andrequires extra processing power. When selecting distant or smallobjects, the existing techniques suffer from low precision and accuracy,making it difficult for the XR system to correctly interpret whichobjects the user intends to select. Also, when the user intends toselect objects that, from the XR system's viewpoint, occlude each other,the XR system does not know how to disambiguate which object(s) the userintends to select. Furthermore, with existing XR systems, when theobject is within a dense field of other objects, focused refinement ofthe user's intent within the object group is impossible.

Technologies for overcoming these drawbacks in existing systems includesystems and processes to identify and interpret “cut,” “lasso” and“frame” gestures for selecting multiple objects, a “cut-click” gesturefor activating controls, and various two-handed “bimanual” gestures todisambiguate between potential target objects.

FIG. 19 is a flow diagram illustrating a process 1900 used in someimplementations of the present technology for selecting objects using acut gesture. At block 1902, process 1900 can identify an actioncorresponding to starting object selection. In some implementations, theaction can be a particular gesture, such as a “pinch” between the user'sthumb and index or middle finger. In some implementations, the actioncan also include identifying gating conditions such as a hand trackingsystem providing one or more of indications that the user's hand isin-view, that a confidence value in the current hand tracking positionis above a threshold, or that the hand is not in a resting state.

At block 1904, process 1900 can continuously determine a control pointand casting direction based on a tracked position of one or more bodyparts. The body part(s), for example, can be part of a hand (e.g.,fingertips, wrist, palm, fist, etc.), an eye, a head, a chest or torso,a leg or foot, etc. The control point can be at one of the body parts oroffset from it. In some implementations, the casting direction can bealong a line connecting one of the user's eyes to the control point. Forexample, process 1900 can record intersections between objects and aline that extends outward from the control point and that intersectsboth the control point and the user's dominant eye. The effect of thisexample appears to the user as though they are “drawing” in space bymoving the control point. In other implementations, projections asdiscussed above in relation to FIG. 5 through FIG. 11 can be used. Inthese implementations, the casting direction can be along a line thatpasses through the control point and an origin point, such as ashoulder, eye, or hip. In some implementations, the drawing can be doneby a user's gaze, where the control point can be an estimated positionof a center of a user's pupil and the origin point can be an estimatedposition of the center of a user's retina.

At block 1906, process 1900 can display a path based on the castingdirection. The path can be a record of locations indicated by thecontrol point and casting direction. In some implementations, the pathcan appear to the user as if she is drawing the path in the artificialreality environment by moving the control point. In otherimplementations, the path can appear to the user as if she is creatingthe path at the end of a projection. In some implementations, process1900 does not display a path, skipping block 1906.

At block 1908, as process 1900 continuously monitors the control pointand casting direction, process 1900 can record any objects, from theuser's point of view, that the control point intersects. In someimplementations, process 1900 can record indications of any object thata cast projection intersects.

At block 1910, in response to one or more triggers such as the userreleasing a gesture or the velocity of the user's hand movement fallingbelow a threshold, process 1900 can select the objects for whichindications were recorded at block 1908. For example, process 1900 couldhave started at block 1902 by recognizing that a user formed a gesturebringing all five fingertips together. At blocks 1904, 1906, and 1908,process 1900 could have monitored a point just in front of the user'sfingertips as a control point, determined a direction based on a trackedposition of the user's dominant eye, and recorded each object that thecontrol point intersected from the user's point of view. Process 1900could have continued this until the user released the five-fingergesture, at which point process 1900 can cause any recorded objects tobe selected.

FIG. 20 is a conceptual diagram illustrating an example 2000 ofselecting objects using a cut gesture. In example 2000, the user beganby making a gesture 2002, bringing her thumb-tip, index fingertip, andmiddle fingertip together. The hand interaction system then begantracking control point 2004 (a point offset in front of the gesture2002) and casting direction 2006. In some implementations, the user canperform distinct gestures—a first (e.g., bringing thumb and middlefingertip together) to initiate casting, and then a second (e.g.,bringing thumb and index fingertip together) to begin selecting objectsthat intersect with the projection. In example 2000, casting direction2006 is formed by determining the line connecting the user's eye (notshown) and the control point 2004. Thus, from the user's point of view,control point 2004 occludes drawing point 2008. When the user firstformed the gesture 2002, the drawing point 2008 was at point 2010. Theuser then moved her hand, causing the control point 2004 to move, whichin turn caused drawing point 2008 to progress from point 2010 along path2012. As a result, path 2012 intersects with objects 2014 and 2016, butdoes not intersect with object 2018, causing the hand interaction systemto record indications of objects 2014 and 2016. In example 2000, thehand interaction system displays path 2012 in the artificial realityenvironment for the user to see. When the user releases gesture 2002,the hand interaction system selects objects 2014 and 2016, based on therecorded indications. As shown in the figures herein, objects areillustrated as “selected” when the wireframe of that object has boldlines, as compared to other objects in that figure.

FIG. 21 is a flow diagram illustrating a process 2100 used in someimplementations of the present technology for actuating objects using acut-click gesture. At block 2102, process 2100 can identify an actioncorresponding to starting object selection. In some implementations, theaction can be a particular gesture, such as a “pinch” between the user'sthumb and index or middle finger. In some implementations, the actioncan also include identifying gating conditions such as a hand-trackingsystem providing one or more indications that the user's hand is inview, that a confidence value in the current hand-tracking position isabove a threshold, or that the hand is not in a resting state.

At block 2104, process 2100 can continuously determine a control pointand casting direction based on a tracked position of one or more bodyparts. The body part(s), for example, can be part of a hand (e.g.,fingertips, wrist, palm, fist, etc.) part of an eye, a head, a chest ortorso, a leg or foot, etc. The control point can be at one of the bodyparts or offset from it. In some implementations, the casting directioncan be along a line connecting one of the user's eyes to the controlpoint. For example, process 2100 can record intersections betweenobjects and a line that extends outward from the control point and thatfalls along a line that intersects both the control point and the user'sdominant eye. The effect of this example appears to the user as thoughthey are “drawing” in space by moving the control point. In otherimplementations, projections as discussed above in relation to FIG. 5through FIG. 11 can be used. In these implementations, the castingdirection can be along a line that passes through the control point andan origin point, such as a shoulder, eye, or hip. In someimplementations, the drawing can be done by a user's gaze, where thecontrol point can be an estimated position of a center of a user's pupiland the origin point can be an estimated position of the center of auser's retina.

At block 2106, process 2100 can display a path based on the castingdirection. The path can be a record of locations indicated by thecontrol point and casting direction. In some implementations, the pathcan appear to the user as if she is drawing the path in the artificialreality environment by moving the control point. In otherimplementations, the path can appear to the user as if she is creatingthe path at the end of a projection. In some implementations, process2100 does not display a path, skipping block 2106.

At block 2108, as process 2100 continuously monitors the control pointand casting direction, process 2100 can record any objects that thecontrol point intersects from the user's point of view, which is basedon the casting direction. In some implementations where a projection isused, process 2100 can record indications of any object that theprojection intersects. In some implementations, instead of or inaddition to recording intersections, whenever process 2100 identifies anintersection with an actionable object (such as a button or othercontrol), process 2100 can actuate the object.

At block 2110, in response to one or more triggers such as the userreleasing a gesture or the velocity of the user's hand movement fallingbelow a threshold, process 2100 can actuate the objects for whichindications were recorded at block 2108. For example, process 2100 couldhave started at block 2102 by recognizing that a user formed a gesturebringing all five finger tips together. At blocks 2104, 2106, and 2108,process 2100 could have monitored a point just in front of the user'sfingertips as a control point, determined a direction based on a trackedposition of the user's dominant eye, and recorded each actionable objectthat, from the user's point of view, the control point intersected.Process 2100 could have continued this until the user released thefive-finger gesture, at which point process 2100 can cause any recordedobjects to be actuated. In some implementations, process 1900 and 2100can be performed together where movement of the control point can causeactionable intersected objects to be actuated and other intersectedobjects to be selected. In some implementations, the actionable objectsare actuated, and both the actionable and non-actionable objects areselected.

FIG. 22 is a conceptual diagram illustrating an example 2200 ofactuating a button object using a cut-click gesture. In example 2200,the user began by making a gesture 2202, bringing her thumb-tip andmiddle fingertip together. The hand interaction system then begantracking control point 2204 (a point offset in front of the gesture2202) and casting direction 2206. In example 2200, casting direction2206 is formed by determining the line connecting the user's eye (notshown) and the control point 2204. Thus, from the user's point of view,control point 2204 occludes drawing point 2208. When the user firstformed the gesture 2202, the drawing point 2208 was at point 2210. Theuser then moved her hand, causing the control point 2204 to move, whichin turn caused drawing point 2208 to progress from point 2210 along path2212. As a result, path 2212 intersects with button object 2214, causingthe hand interaction system to actuate button object 2214. In example2200, the hand interaction system displays path 2212 in the artificialreality environment.

FIG. 23 is a flow diagram illustrating a process 2300 used in someimplementations of the present technology for selecting objects using alasso gesture. At block 2302, process 2300 can identify an actioncorresponding to starting object selection. In some implementations, theaction can be a particular gesture, such as a “pinch” between the user'sthumb and index or middle finger. In some implementations, the actioncan also include identifying gating conditions such as a hand-trackingsystem providing one or more indications that the user's hand is inview, that a confidence value in the current hand-tracking position isabove a threshold, or that the hand is not in a resting state.

At block 2304, process 2300 can continuously determine a control pointand casting direction based on a tracked position of one or more bodyparts. The body part(s), for example, can be part of a hand (e.g.,fingertips, wrist, palm, fist, etc.) part of an eye, a head, a chest ortorso, a leg or foot, etc. The control point can be at one of the bodyparts or offset from it. In some implementations, the casting directioncan be along a line connecting one of the user's eyes to the controlpoint. For example, process 2300 can determine whether objects fallwithin a distorted cone formed with the distorted cone's tip at theuser's eye and the shape of the distorted cone's sides based on the usermoving the control point. As used herein, a “distorted cone” is a conewith a base that is not necessarily a circle or oval shape. In someimplementations, the distorted cone shape can be created by a usermoving her gaze, where the gaze is determined by a line formed from anestimated position of a center of a user's pupil and an estimatedposition of the center of a user's retina.

At block 2306, process 2300 can display a path based on the castingdirection. The path can be a record of locations indicated by thecontrol point and casting direction. In some implementations, the pathcan appear to the user as if she is drawing the path in the artificialreality environment by moving the control point. In someimplementations, process 2300 does not display a path, skipping block2306.

At block 2308, as process 2300 continuously monitors the control pointand casting direction, process 2300 can determine that the user movementof the control point was intended as a lasso input. In someimplementations, process 2300 makes this determination by providingindications of the path to a machine learning model trained to identifylasso inputs. In some implementations, process 2300 makes thisdetermination by flattening the path, from the user's perspective, to a2D plane and determining that the path intersects itself. The lassoinput can define a closed outline of a shape formed by the lasso input.If the lasso input does not make a closed shape, process 2300 can closeit with a straight line or a curve matching the lasso input.

At block 2310, in response to one or more triggers such as theidentification of the lasso input, the user releasing a gesture, or thevelocity of the user's hand movement falling below a threshold, process2300 can identify a distorted cone formed with the tip of the distortedcone at one of the user's eyes and the distorted cone shape formed bythe closed shape defined by the lasso input. For example, the distortedcone can be formed by each possible line that extends from the user'seye and that passes through a point on the closed shape.

At block 2312, process 2300 can select the objects which fall within thedistorted cone identified at block 2310. In some implementations,process 2300 can also select objects that at least partially intersectthe distorted cone or have a threshold amount of their volume within thedistorted cone (e.g., at least 30%, 50%, or 75%). For example, process2300 could have started at block 2302 by recognizing that a user formeda gesture bringing all five finger tips together. At blocks 2304-2310,process 2300 could have monitored a point just in front of the user'sfingertips as a control point, determined a direction based on a trackedposition of the user's dominant eye, tracked a path defined by thecontrol point from the user's point of view, and used a machine learningmodel to determine that the path formed a lasso input. Process 2300could then identify a distorted cone defined by a path of a line thatbegins at the user's eye and that is moved to also extend through everypoint on a closed portion of the lasso input. Process 2300 could thencause any objects that at least partially fall within the distorted coneto be selected.

In some implementations, multiple of the selection processes describedherein can be combined into a single gesture. For example, processes1900 and 2300 can be combined. In one example, process 1900 can begin atblock 1902 while continuously performing the analysis from block 2308 todetermine if the input is a lasso gesture. If a lasso gesture isidentified, process 1900 can transition to block 2308 of process 2300.if no lasso gesture is identified, process 1900 can continue. In anotherexample, a lasso gesture drawn by the user can have a closed shape witha beginning and/or an ending “tail.” This example combination ofprocesses 1900 and 2300 can select all objects that intersect the pathof the lasso gesture (including the tail) from the user's point of viewand can select all objects that fall inside the distorted cone formed bythe closed shape portion of the lasso gesture. In another example,process 2100 can be combined with process 2300 such that any actionableobject selected by process 2300 can also be actuated.

FIG. 24A is a conceptual diagram illustrating an example 2400 ofselecting objects using a lasso gesture. In example 2400, the user beganby making a gesture 2402, bringing her thumb-tip, index fingertip, andmiddle fingertip together. The hand interaction system then begantracking control point 2404 (a point offset in front of the gesture2402) and a casting direction formed by determining the line connectingthe user's eye (not shown) and the control point 2404. Thus, from theuser's point of view, moving control point 2404 can be used to “draw” apath in the artificial reality environment. When the user first formedthe gesture 2402, she began drawing at point 2406. The user then movedher hand, causing the control point 2404 to move, creating path 2408.When path 2408 intersects with itself (from the user's point of view) atpoint 2410, the hand interaction system determined, based on a machinelearning model, that the user has drawn a lasso selection. The handinteraction system then formed the closed section of path 2408 into adistorted cone (not shown) and determined that the distorted cone atleast partially intersects with objects 2412, 2414, and 2416. The handinteraction system thus selected the intersected objects 2412, 2414, and2416.

FIG. 24B is another conceptual diagram illustrating an example 2450 ofselecting objects using a lasso gesture. In example 2450, the userformed a path with a closed shape 2452 by moving a control point basedon a gesture in a manner similar to example 2400. The hand interactionsystem formed a distorted cone 2454 with the tip of the cone at theuser's dominant eye 2456 and extending so that the edges of thedistorted cone coincide with closed shape 2452. The hand interactionsystem then determined any objects that are beyond the shape 2452 (i.e.on the opposite side of the closed shape 2452 from the user) and thatintersect with that portion of the distorted cone 2454. In example 2450,these are objects 2456 and 2458. The hand interaction system thusselected the intersected objects 2456 and 2458, but not non-intersectedobjects 2460 and 2462.

FIG. 25 is a flow diagram illustrating a process 2500 used in someimplementations of the present technology for selecting objects using aframe gesture. At block 2502, process 2500 can identify an actioncorresponding to starting object selection. In some implementations, theaction can be a two-handed “pinch” gesture, formed with the user's thumbfrom each hand touching the index or middle finger of that hand, withthe two pinches touching at the thumb/finger intersection point. In someimplementations, the action can be a two-handed “L” gesture, formed witha first hand having the thumb sticking up perpendicular to the floor andthe index finger parallel to the floor and with a second hand having thethumb sticking down perpendicular to the floor and the index fingerparallel to the floor, defining two opposite corners of a rectangle. Insome implementations, the gesture can be the user's fingers on one orboth hands defining another shape, such as a triangle or oval. In someimplementations, the gesture can be formed with one hand. For example,the gesture can be a pinch that, when opened, defines opposite points ofa rectangle corresponding to points at the tips of the user's thumb andfinger. As another example, the gesture can be a pinch that begins thesystem tracking a control point. The control point can be moved to drawa line, defining opposite points of a rectangle as the start and endpoints of the line. In some implementations, the action can also includeidentifying gating conditions such as a hand-tracking system providingone or more indications that the user's hand is in view, that aconfidence value in the current hand-tracking position is above athreshold, or that the hand is not in a resting state.

At block 2504, process 2500 can continuously determine a shape definedby a first tracked portion of a first user hand and a second trackedportion of a second user hand. The shape can be a rectangle and thetracked hand portions can be points identified at block 2502corresponding to opposite corners of the rectangle. As the user movesher hands, process 2500 can determine new sizes and positions for therectangle. In other implementations, the shape can be defined byascribing lines or curves to portions of a user's hand, connecting thelines and/or snapping the lines to conform to a given shape. Forexample, process 2500 can be configured to recognize a rectangle, andthe user can make an L shape with each hand. Process 2500 can ascribe aline to each finger making part of the L shape, extend them tointersect, and snap these four lines to form right angles, making arectangle. As another example, process 2500 can be configured torecognize an oval, and the user can make a “C” shape with the thumb andindex finger of one hand and another backward “C” shape with the thumband index finger of the other hand. Process 2500 can ascribe a curve toeach C shape, extend them to intersect, and smooth these curves to forman oval. As discussed above, in some implementations, the shape can bedefined by a gesture using a single hand, such as a rectangle formedwith opposite corners defined at points on a user's index finger andthumb or based on a line drawn by a user (in a manner similar to thepath tracked by blocks 1902 and 1904) that defines opposite corners of arectangle as the start and end points of the line.

At block 2506, process 2500 can display a representation (such as anoutline) of the shape determined at block 2504. For example, if the useris making a gesture to pull apart opposite corners of a rectangle usingpinch gestures, a rectangle can be continuously drawn showing the useran outline of the rectangle. In some implementations, process 2500 doesnot display a shape representation, skipping block 2506.

At block 2508, process 2500 can identify a pyramid formed with the tipof the pyramid at one of the user's eyes and the pyramid walls beingformed based on the rectangle determined at block 2504. For example, thepyramid can be formed by the four triangles that extend from the user'seye and that pass through one edge of the rectangle. In someimplementations where the shape determined at block 2504 is other than arectangle, other 3D shapes can be determined, such as a tetrahedron ifthe shape is a triangle or a cone if the shape is an oval (formedsimilarly to block 2310).

At block 2510, in response to one or more triggers such as continuouslyfollowing the identification of the action at block 2502, upon the userreleasing the gesture identified at block 2502, upon recognizing anotherhand gesture (e.g., thumb to middle fingertip to trigger frame drawingand thumb to index fingertip to select objects within the pyramid), orwhen the velocity of the user's hand movement falls below a threshold,process 2500 can select the objects that fall within the pyramid (orother shape) identified at block 2508. In some implementations, process2500 can also select objects that at least partially intersect thepyramid (or other shape) or have a threshold amount of their volumewithin it (e.g., at least 20%, 50%, or 85%). For example, process 2500could have started at block 2502 by recognizing that a user formed twotouching pinch gestures. At blocks 2504-2510, process 2500 could havecontinuously determined the size and position of the rectangle formedwith opposite corners at the user's two pinch gestures as the user pullsthem apart. Process 2500 could then have drawn this rectangle for theuser to see, identified a pyramid with a tip at the user's dominant eyeand with the pyramid shape defined by the rectangle, and caused anyobjects that at least partially fall within the pyramid to be selected.

In some implementations, multiple of the selection processes describedherein can be combined into a single gesture. For example, processes2100 and 2500 can be combined such that any actionable object selectedby process 2500 can also be actuated.

FIG. 26A is a conceptual diagram illustrating an example 2600 ofselecting objects using a frame gesture. In example 2600, the user beganby making gestures 2602A and 2602B, bringing her thumb-tips and indexfingertips together and touching those gestures together at point 2612.The hand interaction system then began tracking opposite corners ofrectangle 2604 based on the locations of gestures 2602A and 2602B, fromthe user's point of view. Thus, from the user's point of view, movinggesture points 2602A and 2602B formed rectangle 2604. As the user movedher hands apart, the corners of rectangle 2604 moved apart, increasingthe size of rectangle 2604. As the user formed this rectangle, the handinteraction system determined a pyramid formed with the pyramid tip atthe user's dominant eye and with sides extending through the edges ofrectangle 2604. In example 2600, the hand interaction systemcontinuously selected (or deselected) objects that at least partiallyintersect with the pyramid until the user released one of gestures 2602Aor 2602B. The hand interaction system thus selected the intersectedobjects 2606 and 2608, but not 2610.

FIG. 26B is another conceptual diagram illustrating an example 2650 ofselecting objects using a frame gesture. In example 2650, the userformed a rectangle 2652 by forming two pinch gestures and pulling themapart in a manner similar to example 2600. The hand interaction systemformed a pyramid 2654 with the tip of the pyramid at the user's dominanteye 2656 and extending so that the four triangles that form edges of thepyramid coincide with rectangle 2652. The hand interaction system thendetermined any objects that are both beyond the rectangle 2652 (i.e. onthe opposite side of the rectangle 2652 from the user) and that fallcompletely within the pyramid 2654. In example 2650, this is object2658. The hand interaction system thus selected the enveloped object2658, but not non-enveloped objects 2660 and 2662.

Bimanual Interactions

Single hand gestures limit how expressive a user can be when interactingwith an XR system. The hand interaction system can track postures ofboth user hands, allowing a second hand (e.g., a non-dominant hand) toprovide nuances for a gesture controlled by a first hand (e.g., adominant hand). These bimanual (i.e., two-handed) interactions allowusers to transfer theft everyday experiences and skills to interactionsin the artificial reality environment. For example, when performing aninteraction with a projection casting technique discussed above, theprojection can intersect with more or fewer objects than the userintends, such as when multiple objects occlude one another, when objectsare tightly grouped and a user wants to select a subset of them, or whenobjects are spread apart and the user wants to select all of them. Inthese cases, a second gesture performed by a second hand (and/or withdifferent parts of the same hand) can control aspects of the projectionsuch as the length of a ray projection (e.g., a “hook” interaction) orthe diameter of a cone projection (e.g., a “trombone” interaction). Insome implementations, the user may want to perform more complicatedinteractions with a target object or objects and can specify details ofthese interactions by interacting with a control using her second hand.

FIG. 27 is a flow diagram illustrating a process 2700 used in someimplementations of the present technology for disambiguating betweentarget and non-target objects that intersect with a ray projection. Forexample, this can allow a user to disambiguate objects that occlude oneanother when using a ray projection.

At block 2702, process 2700 can determine a control point and castingdirection based on a tracked position of one or more body parts. Process2700 can accomplish this in a manner similar to block 1202. At block2704, process 2700 can generate a ray projection from the control pointalong the casting direction. Process 2700 can accomplish this in amanner similar to block 1204.

At block 2706, process 2700 can continuously determine a distancerelationship between a first hand (e.g., a dominant hand) controllingthe control point and a second hand (e.g., a non-dominant hand). In someimplementations, instead of the distance between two hands, the distancerelationship can be based on the distance between two fingers (e.g., tipof index finger and tip of thumb) on one hand. In other implementations,the distance relationship can be based on the distance between one handand another body part, such as the user's torso. In someimplementations, the ray projection can be along a user's gaze, and theuser can specify a distance by moving her hand closer or further fromher eye along the ray. In various implementations, the distance can bein any direction or can be in a particular direction (e.g.,perpendicular to the ray or along the ray).

In various implementations, the distance relationship can be linearly orexponentially proportional to the actual distance between the user'shands. In some implementations, this relationship can be based on aspeed at which the user changes the distance between her hands. Forexample, an amount that process 2700 determines can be increased ordecreased logarithmically such that faster movements increase the amountmore than slow movements. This allows the user to use fast movements tocontrol the amount with larger granularity and slower movements to makemore precise adjustments. In some implementations, the distancerelationship can be scaled according to a determined size of theartificial reality environment. For example, in smaller spaces movementscan be more precise, while in larger spaces movements can extend the rayfurther to interact with objects at a greater distance.

At block 2708, process 2700 can continuously set a length of the rayprojection or a “hook” location based on the distance relationshipdetermined at block 2706. In some implementations, instead of setting aray length, process 2700 can set an interaction point along the raybased on the distance relationship. Whether at the end of the ray or ata point along the ray, this interaction point is referred to herein asthe “hook.” In some implementations, process 2700 can provide a visualrepresentation of the ray and can include a graphical representation(such as a sphere) of the hook.

At block 2710, process 2700 can identify one or more objects based on anintersection with the hook. In some implementations, this can occur assoon as the hook intersects with an object. In other implementations,process 2700 can identify a third gesture performed by the user, such asmaking a click gesture with one hand, nodding her head, tapping herfoot, etc., as the trigger to select object(s) that the hook intersectswhen the gesture is made. At block 2712, process 2700 can toggleselection of, or otherwise interact with, the object(s) identified atblock 2710.

FIGS. 28 and 29 are conceptual diagrams illustrating an example 2800 and2900 of using a hook to disambiguate between selected and non-selectedobjects that intersect with a ray projection 2802. Example 2800 beginswith a user creating ray projection 2802 by performing pinch gesture2804 between her thumb and middle finger on her dominant hand 2806. Theuser can position the ray 2802 so that it intersects with objects 2808and 2810. The user can then control the length of the ray, with a hook2814 at the end of the ray 2802, based on a distance 2816 between herdominant hand 2806 and non-dominant hand 2818. As the hook 2814intersects with the object 2810, which the user intends to target, shecan make a second pinch gesture (not shown), this time with her indexfinger and thumb on her dominant hand 2806. As the hand interactionsystem identifies this gesture, it selects object 2810, which the hook2814 intersects at that moment. In example 2900, the user then changesthe length of ray 2802, and accordingly the position of the hook 2814,by lengthening the distance 2816 between her dominant hand 2806 and hernon-dominant hand 2818, while still holding the firstthumb/middle-finger pinch, causing the hook 2814 to intersect withobject 2812. The user can also select object 2812, now that itintersects with the hook 2814, by again making the gesture 2902 with afurther thumb/index-finger pinch.

FIG. 30 is a flow diagram illustrating a process 3000 used in someimplementations of the present technology for disambiguating, with a“trombone” gesture, between target and non-target objects that intersectwith a cone or cylinder projection. For example, this can allow a userto expand or contract the diameter of a cone or cylinder projection toeasily control which objects the cone or cylinder intersects.

At block 3002, process 3000 can determine a control point and castingdirection based on a tracked position of one or more body parts. Process3000 can accomplish this in a manner similar to blocks 1402 or 1602. Atblock 3004, process 3000 can generate a cone or cylinder projection fromthe control point along the casting direction. Process 3000 canaccomplish this in a manner similar to blocks 1404 or 1604.

At block 3006, process 3000 can continuously determine a distancerelationship between a first hand (e.g., a dominant hand) controllingthe control point and a second hand (e.g., a non-dominant hand). In someimplementations, instead of the distance between two hands, the distancerelationship can be based on the distance between two fingers (e.g., tipof index finger and tip of thumb) on one hand. In other implementations,the distance relationship can be based on the distance between one handand another body part, such as the user's torso. In someimplementations, the cone or cylinder projection can be along a user'sgaze, and the user can specify a distance by moving her hand closer orfurther from her eye along the center of the cone or cylinder. Invarious implementations, the distance can be in any direction or can bein a particular direction (e.g., perpendicular or parallel to the centerof the cone or cylinder).

In various implementations, the distance relationship can be linearly orexponentially proportional to the actual distance between the user'shands. In some implementations, this relationship can be based on aspeed at which the user changes the distance between her hands. Forexample, an amount that process 3000 determines can be increased ordecreased logarithmically such that faster movements increase the amountmore than slow movements. This allows the user to use fast movements tocontrol the amount with larger granularity and slower movements to makemore precise adjustments.

At block 3008, process 3000 can continuously set a diameter of thecylinder or of the base of the cone based on the distance relationshipdetermined at block 3006. In some implementations, process 3000 canprovide a visual representation of the cone or cylinder, showing itchange size as the user changes the distance relationship.

At block 3010, process 3000 can identify one or more objects based on atleast partial intersection or full encompassment by the cone orcylinder. In some implementations, this can occur as soon as the cone orcylinder intersects with an object. In other implementations, process3000 can identify a third gesture performed by the user, such as makinga click gesture with one hand, nodding her head, tapping her foot, etc.,as a trigger to identify any object that the cone or cylinder intersectsat that moment. At block 3012, process 3000 can toggle selection of, orotherwise interact with, the object(s) identified at block 3010.

In an alternative implementation that combines processes 2700 and 3000,the user can control the length of the cylinder or cone with one gesture(e.g., the distance between the tip of a thumb and forefinger on adominant hand) and can control the diameter of the cylinder or cone basewith another gesture (e.g., the distance between her two hands). As amodification of this alternative, instead of a cylinder or coneprojection, a sphere projection can be used, where the user controls thelocation of a sphere along a ray based on one gesture (e.g., thedistance between her two hands) and the diameter of the sphere withanother gesture (e.g., the distance between the tip of a thumb andforefinger on a dominant hand).

FIGS. 31 and 32 are conceptual diagrams illustrating examples 3100 and3200 of trombone gestures for disambiguating between selected andnon-selected objects that intersect with a cone projection. Example 3100begins with a user creating cone projection 3102 by performing pinchgesture 3104 between her thumb and middle finger on her dominant hand3106. The user can position the cone 3102 so that it intersects withobjects 3108, 3110, 3112, and 3120. The user can then control thediameter of the base 3114 of cone 3102 based on a distance 3116 betweenher dominant hand 3106 and non-dominant hand 3118. The hand interactionsystem selects objects 3108, 3110, 3112, and 3120 due to theirintersection with cone 3102. In example 3200, the user has changed thediameter of the base 3114 of cone 3102 by changing distance 3116 betweenher dominant hand 3106 and non-dominant hand 3118, while still holdingthe first thumb/middle-finger pinch gesture, causing the cone 3102 tointersect with only objects 3112 and 3108. In response, the handinteraction system can deselect objects 3120 and 3110, but keep objects3108 and 3112 selected. The user can continue until her intended targetobjects are selected, at which point she releases gesture 3104.

FIGS. 33 and 34 are conceptual diagrams illustrating examples 3300 and3400 of a trombone gesture to disambiguate between selected andnon-selected objects that intersect with a cylinder projection. Example3300 begins with a user creating cylinder projection 3302 by performingpinch gesture 3304 between her thumb and middle finger on her dominanthand 3310. The user can position the cylinder 3302 so that it intersectswith objects 3308, 3306, and 3330. The user can then control thediameter of the base 3314 of cylinder 3302 by changing the distance 3316between her dominant hand 3310 and non-dominant hand 3318. The handinteraction system selects objects 3308, 3306, and 3330, as these arethe objects that intersect with cylinder 3302, but does not selectobject 3332 because it does not intersect with cylinder 3302. In example3400, the user then changes the diameter of the base 3314 of cylinder3302 by changing distance 3316 between her dominant hand 3310 andnon-dominant hand 3318, while still holding the firstthumb/middle-finger pinch. This change causes the cylinder 3302 tointersect with objects 3306, 3308, 3330, and 3332. The user can continueuntil her intended target objects are selected, at which point shereleases gesture 3304.

While combinations of gestures, as discussed above, are powerfulinteraction techniques, in some circumstances mapping a differentgesture to every action can be confusing for a user or limiting in whichactions a user can perform. However, when interacting with distantobjects, menus attached to those objects can be difficult for a user tointeract with. Menus that are world-locked can be limiting where a userwants to move herself or a target object while interacting with themenu. It can also be difficult for a user to match world-locked menus totarget objects. These problems can be overcome with a menu that relatesto an object selected with a projection, but that is attached to theprojection casting hand instead of the object. This allows a second handto interact with the menu, and thereby the object, with ease. FIG. 35 isa flow diagram illustrating a process 3500 used in some implementationsof the present technology for a bimanual interaction with an objectusing a menu attached to a dominant hand.

At block 3502, process 3500 can receive an identification of one or moreobjects selected using a dominant hand. In some implementations, theselection can be performed using one of the projection ormulti-selection processes described above. In some implementations, theselection can be associated with a particular gesture, such as a gestureused to initiate a projection or selection, and process 3500 cancontinue only while the user holds that gesture. For example, a user cancast a ray projection to select an object, and process 3500 can continueproviding a menu corresponding to that selected object until the usermoves the projection (unselecting the object) or releases the projectiongesture.

At block 3504, process 3500 can obtain one or more menus associated withthe one or more objects for which process 3500 received indications atblock 3502. At block 3506, process 3500 can display one or more menus inassociation with a dominant hand. In other implementations, instead ofalways displaying the one or more menus in association with the dominanthand, process 3500 can display the one or more menus in association withthe non-dominant hand or with a hand that performed the selection of theone or more objects. In various implementations, each of the one or moremenus can be offset from a point on the hand (such as the wrist, centerof the thumb, at a fingertip, middle of the back of the hand, center ofmass of the hand, etc.), overlaid as a “skin” at such a point, wrappedaround a part of the hand (e.g., to appear as a bracelet, ring, etc.),or combinations thereof. In some implementations, the one or more menuscan be triggered to appear and disappear based on a gesture, such asmovement of a particular finger or rotation of the hand.

At block 3508, process 3500 can receive interactions with the displayedone or more menus. In some implementations, the interaction can occurwith the same hand with which the menu is associated (e.g., a “ring”menu appearing on the middle finger can be tapped by a thumb on thatsame hand) or can be an interaction from the opposite hand, whether ornot it is non-dominant. The one or more menus can have controls toperform any type of interaction with a selected object, and theinteraction with the menu can trigger that interaction. In someimplementations, the menu selection can cause the hand interactionsystem to interpret further gestures in a particular way. For example,if a menu selection is “move,” then movement of the dominant hand cancause the hand interaction system to move the selected one or moreobjects accordingly. As another example, if a menu selection is“rotate,” then rotation of the dominant hand can cause the handinteraction system to rotate the selected one or more objectsproportionally.

FIGS. 36, 37 and 38 are conceptual diagrams illustrating examples 3600,3700 and 3800 of a bimanual interaction with an object using a menuattached to a dominant hand. In example 3600, an indication of object3602 is received due to the selection of object 3602 using rayprojection 3604, cast by the user's dominant hand 3606. In example 3700,in response to the selection of object 3602, menu 3702 appears inassociation with dominant hand 3602, offset from a center of mass fromthe dominant hand 3602. Menu 3702 includes controls for object 3602 thatcan be actuated by the user's non-dominant hand 3710. In example 3700,the controls include control 3704 for triggering simultaneous XYZ-axismovement of object 3602 (thereby allowing the user to use menu 3702 todisambiguate translation, scale, and rotation operations for moreprecise manipulation in 3D space), control 3706 for allowing freerotation of the selected object 3602 according to rotation of thedominant hand 3606, and control 3708 for resizing the selected object3602. In example 3800, the user has actuated control 3706 by performingpinch gesture 3802, using her non-dominant hand 3710, next to control3706. The actuation of control 3706 causes the hand interaction systemto map rotation of the dominant hand 3606 to rotation of the selectionobject 3602.

Operational Modes

There are different types of interactions that a user can perform inrelation to an object, such as moving an object within an artificialreality environment and actuating controls on the object. However, thegestures to perform such interactions can be ambiguous, causing existingXR systems to incorrectly distinguish between them. Especially asprecision degrades over distance, users of existing XR systems findinteractions tiring and stressful due to these systems' use of the sameinput modality in different situations and their failure to distinguishbetween interaction types. To address these issues, the hand interactionsystem can switch operational modes (e.g., between global and localinteraction modes). Global mode can be used to manipulate objects withinan environment (e.g., selecting objects, controlling object placement,orientation, size, etc., in the virtual environment) or to switch to aninput modality useful for such global interactions (e.g., ray, sphere,or cone casting). Local mode can be used to interact with interfaces ofan object (e.g., pressing buttons on an object or interacting with menusassociated with the object) or to switch to an input modality useful forsuch local interactions (e.g., mapping buttons to user fingers). Thus,ambiguous input can be interpreted according to whichever mode iscurrently set for the hand interaction system. In variousimplementations, alternate or additional operations modes can be used.For example, more than two modes can be defined, modes can be definedfor different purposes, modes can be triggered in different ways, etc.

In various implementations, the user can explicitly select betweenoperational modes (e.g., with a gesture or button press) or modeselection can be implicit (e.g., based on user gaze). In response to anexplicit or implicit trigger corresponding to a switch betweenoperational modes, the hand interaction system can then interpret inputgestures according to a mapping of gestures to actions defined for theselected mode. For example, in response to identifying a switch to afirst mode (e.g., global mode) the hand interaction system can interpreta first gesture by mapping the first gesture to a first action in afirst set of actions defined for the first (e.g., global) mode. Then, inresponse to a second trigger causing a switch to a second operationalmode (e.g., local mode), the hand interaction system can interpret asecond gesture by mapping the second gesture to a second action in asecond set of actions defined for the second (e.g., local) mode.

In some implementations, an input modality can change based on whichoperational mode is activated. For example, in global mode, inputs canbe switched to an input modality based determining gesture direction(e.g., projection casting, such as ray, cylinder, sphere, or coneprojections). In local mode, inputs can be switched to other modalities,such as by mapping controls associated with selected objects to fingerson the user's hand, where a particular detected movement of the mappedfinger controls actuation of the corresponding control.

FIG. 39 is a flow diagram illustrating a process 3900 used in someimplementations of the present technology for switching input modalities(e.g., for global or local mode interactions). At block 3902, process3900 can identify a trigger for transitioning between operational modes,and at block 3904 process 3900 can identify which mode the triggercorresponds to (e.g., whether the trigger is to enter global or localinput mode). In some implementations, the trigger can be an explicituser action. As examples, a particular gesture can cause the handinteraction system to toggle between modes or to switch to a first(e.g., local) mode while the gesture is held and switch to a second(e.g., global) mode when the gesture is not identified; the handinteraction system can switch to the first mode when an input hand isrotated vertical and switch to the second mode when not; the handinteraction system can toggle between modes in response to a real orvirtual button press or a vocal command; etc. In some implementations,the trigger to switch modes can be implicit. As an example, the handinteraction system can enter local mode when a user's tracked gazelingers on an object for above a threshold amount of time (e.g., one,two, or three seconds) and can enter global mode when the user's gaze isoff that object for another threshold amount of time. In anotherexample, the hand interaction system can start in global mode, enterlocal mode upon a user first selecting an object, and transition back toglobal mode upon a user tapping off the object or otherwise performing adefined “back” gesture. In some implementations where the trigger is arelational gesture such as rotating the user's hand, hysteresis can beapplied to prevent jitter between modes when the user's hand is on theborder between modes. If the trigger was to transition to a first (e.g.,global) mode, process 3900 can continue to block 3906, and if thetrigger was to transition to second (e.g., local) mode, process 3900 cancontinue to block 3908.

At block 3906, process 3900 can switch to an input modality establishedfor the current mode (e.g., global mode). In some implementations, aninput modality for the global mode can be based on determining gesturedirection (e.g., casting a projection (e.g., ray, sphere, cylinder, orcone), as discussed above). In other implementations, the input modalitycan be presenting a set of controls for global interaction when anobject is selected (e.g., to rotate, resize, or move the object).

At block 3908, process 3900 can switch to an input modality establishedfor a second mode (e.g., local mode). In various implementations, theinput modality can be a mapping of available interactions to movement ofparticular fingers or other gestures; switching between casting types,such as switching from cone casting to ray casting; or displayingcontrols configured for performing location interactions. In someimplementations, local interactions can be for a particular selectedobject, such as one or more objects selected in global mode. This allowsthe user to perform interactions in relation to the selected object(s)without necessarily pointing at or otherwise directing her interactionto the target object. In some cases, the selected object can changewithout exiting the current (e.g., local) mode, e.g., based on theuser's gaze. In other implementations, local interactions can be basedon which object a user indicates.

In some implementations, process 3900 can be performed separately foreach hand. For example, the user's dominant hand can be in a first(e.g., global) mode, controlling a projection for selecting object(s),while the user's non-dominant hand can be in a second (e.g., local)mode, with controls mapped to fingers on the non-dominant hand tointeract with whichever object(s) the dominant hand selects.

After performing block 3906 or 3908, process 3900 can return to block3902 upon identifying another trigger for switching to the alternateglobal or local mode.

FIGS. 40A and 40B are conceptual diagrams illustrating an example 4000of switching input modalities for global or local mode interactions. Inexample 4000, the user transitions between local and global modes basedon the rotational orientation of her hand 4002. In example 4000, theuser has caused the hand interaction system to enter local mode byturning, as indicated by arrow 4004, her hand 4002 vertically (e.g., aplane representing the flat of her hand is at least a threshold numberof degrees, such as 30 degrees, from being perpendicular with anestablished floor plane.)

Upon entering local mode, a set of local mode controls 4008 appear inconjunction with a previously selected object 4006. Alternatively, theobject 4006 can be selected in response to a projection (e.g., a ray)intersecting with the object 4006 when the hand turn occurred. Thecontrols 4008 include buttons 4010-4016. In this example, control 4010is mapped to finger 4018, control 4012 is mapped to finger 4020, control4014 is mapped to finger 4022, and control 4016 is mapped to finger4024. Because the user has extended finger 4018, the correspondingmapped control 4010 is activated.

Example 4000 is continued in FIG. 40B, where the user has caused thehand interaction system to transition to global mode by rotating, asindicated by arrow 4052, her hand 4002 to be horizontal (e.g., the planerepresenting the flat of her hand is at least a threshold number ofdegrees, such as 30 degrees, from being parallel to the establishedfloor plane.) In response, the hand interaction system switches inputmodalities from the hand-mapped controls of FIG. 40A to a ray projection4054, allowing the user to further perform global interactions withobject 4006 or other objects using the ray projection 4054.

FIG. 41 is a flow diagram illustrating a process 4100 used in someimplementations of the present technology for interpreting gestures asparticular interactions in a current (e.g., global or local) mode. Atblock 4102, process 4100 can identify a trigger for transitioningbetween from two modes (e.g., global and local modes), and at block 4104process 4100 can identify which mode the trigger corresponds to (e.g.,whether the trigger is to enter global or local mode). Blocks 4102 and4104 can be performed similarly to blocks 3902 and 3904. If the triggerwas for a first (e.g., global) mode, process 4100 can continue to block4106, and if the trigger was for a second (e.g., local) mode, process4100 can continue to block 4108.

At block 4106, process 4100 can cause the hand interaction system tointerpret further user input according to the current (e.g., global)mode. While at block 4108, process 4100 can cause the hand interactionsystem to interpret further user input according to the second (e.g.,local) mode. Actions that can be taken with respect to an object can bedivided into being performable one of the available modes (e.g., ineither global or local mode). The hand interaction system can match aninput gesture to an action in the set of actions defined for thecurrently activated mode. For example, when in global mode, a gesturedragging downward from an upper right corner of a virtual chess boardcan be interpreted as an intention to rotate the chess board. However,in local mode, the same gesture can be interpreted as moving a rook fromthe upper right corner square forward an amount corresponding to thelength of the gesture. As another example, where a target object is avirtual drawing application, the hand interaction system can interpret aclick-and-drag gesture in local mode as drawing a line within theapplication, while the hand interaction system can interpret the samegesture in global mode as a command to rotate the virtual paneldisplaying the drawing surface. In some implementations, process 4100can be performed separately for each hand. For example, the user'sdominant hand can be in a first (e.g., global) mode, controlling anorientation of a 3D model, while the user's non-dominant can be in asecond (e.g., local) mode to actuate controls on the face of the 3Dmodel.

After performing block 4106 or 4108, process 4100 can return to block4102 upon identifying another trigger for switching to an alternatemode.

FIGS. 42A and 42B are conceptual diagrams illustrating an example 4200of interpreting gestures as interactions in either global or local mode.In example 4200, the user transitions to local mode based on her gazeresting on an object for at least two seconds and transitions to globalmode when her gaze leaves that object for at least 1.5 seconds. Inexample 4200, the user has caused the hand interaction system to enterlocal mode in relation to object 4204 by directing her gaze 4202 atobject 4204 for two seconds. In another implementation, a gesture (e.g.,an index fingertip to thumb gesture) can trigger a local modeinteraction within whichever panel was identified as the target of theuser's gaze when the gesture began. This implicit global selection canbe used in conjunction with the local interaction gesture with nodeliberate user action needed to define global selection.

Example 4200 is continued in FIG. 42B, where the user controls a rayprojection 4254 with her hand 4252. Because the user entered local modebased on her gaze (as shown in FIG. 42A), the ray projection 4254intersecting with the object 4204 causes local interactions by actuatinga control 4256 of object 4204. The user's gaze (not shown in FIG. 42B)has not left object 4204 for more than 1.5 seconds, thus the handinteraction system has remained in local mode. In some implementations,processes 3900 and 4100 can be performed together such that switchingbetween global and local modes can both change an input modality andcause otherwise ambiguous inputs to be interpreted according to acurrent global or local mode.

Reference in this specification to “implementations” (e.g., “someimplementations,” “various implementations,” “one implementation,” “animplementation,” etc.) means that a particular feature, structure, orcharacteristic described in connection with the implementation isincluded in at least one implementation of the disclosure. Theappearances of these phrases in various places in the specification arenot necessarily all referring to the same implementation, nor areseparate or alternative implementations mutually exclusive of otherimplementations. Moreover, various features are described which may beexhibited by some implementations and not by others. Similarly, variousrequirements are described which may be requirements for someimplementations but not for other implementations.

As used herein, a “finger” can be any of the five digits of a hand and afingertip can be a tip of any of the five fingers of a hand.

As used herein, being above a threshold means that a value for an itemunder comparison is above a specified other value, that an item undercomparison is among a certain specified number of items with the largestvalue, or that an item under comparison has a value within a specifiedtop percentage value. As used herein, being below a threshold means thata value for an item under comparison is below a specified other value,that an item under comparison is among a certain specified number ofitems with the smallest value, or that an item under comparison has avalue within a specified bottom percentage value. As used herein, beingwithin a threshold means that a value for an item under comparison isbetween two specified other values, that an item under comparison isamong a middle-specified number of items, or that an item undercomparison has a value within a middle-specified percentage range.Relative terms, such as high or unimportant, when not otherwise defined,can be understood as assigning a value and determining how that valuecompares to an established threshold. For example, the phrase “selectinga fast connection” can be understood to mean selecting a connection thathas a value assigned corresponding to its connection speed that is abovea threshold.

As used herein, the word “or” refers to any possible permutation of aset of items. For example, the phrase “A, B, or C” refers to at leastone of A, B, C, or any combination thereof, such as any of: A; B; C; Aand B; A and C; B and C; A, B, and C; or multiple of any item such as Aand A; B, B, and C; A, A, B, C, and C; etc.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Specific embodiments and implementations have been described herein forpurposes of illustration, but various modifications can be made withoutdeviating from the scope of the embodiments and implementations. Thespecific features and acts described above are disclosed as exampleforms of implementing the claims that follow. Accordingly, theembodiments and implementations are not limited except as by theappended claims.

Any patents, patent applications, and other references noted above areincorporated herein by reference. Aspects can be modified, if necessary,to employ the systems, functions, and concepts of the various referencesdescribed above to provide yet further implementations. If statements orsubject matter in a document incorporated by reference conflicts withstatements or subject matter of this application, then this applicationshall control.

I/We claim:
 1. A computing system comprising: one or more processors;and a memory storing instructions that, when executed by the one or moreprocessors, cause the computing system to perform a process comprising:determining casting direction, from an eye of a user, that is away fromthe user; continuously monitoring a fist control point based on aposition of a part of a first hand of the user; continuously monitoringa second control point based on a position of a part of a second hand ofthe user; defining a rectangle with opposite corners corresponding tothe fist control point and the second control point; generating apyramid projection that extends along the casting direction, wherein atip of the pyramid is at a point corresponding to one of the user's eyesand each of the sides of the pyramid intersect with a different one ofthe lines that forms the rectangle; identifying multiple objects that atleast partially intersect with at least part of the pyramid projection,either simultaneously or at different times; and selecting the multipleobjects.
 2. The computing system of claim 1, wherein the process isperformed in response to identifying that the user performed a gestureof: bringing a thumb-tip on the user's first hand together with afingertip on the user's first hand to make a first contact point;bringing a thumb-tip on the user's second hand together with a fingertipon the user's second hand to make a second contact point; bringing thefirst contact point together with the second contact point; and pullingthe first contact point away from the second contact point.
 3. Thecomputing system of claim 1, wherein the process further comprises: inresponse to the selecting the multiple objects, causing a menu to beattached to the first hand of the user; identifying an interaction, bythe user's second hand, with an item on the menu; and in response to theinteraction, causing a resize or rotation of at least one of theselected multiple objects.
 4. The computing system of claim 1, whereinthe process further comprises: monitoring a rotation of the first handof the user such that rotation of the first hand of the user is mappedto rotation of at least one of the selected multiple objects; and basedon the monitoring, detecting a rotation of the first hand of the userand, in response, causing a corresponding rotation of the at least oneof the selected multiple objects.
 5. The computing system of claim 1,wherein the process further comprises: determining which eye of the useris the user's dominant eye by one or both of: determining which hand theuser employs most often; and/or determining how the user has positionedhimself or herself in relation to past stimulus; wherein the point atwhich the tip of the pyramid is placed is a point corresponding thedetermined dominant eye of the user.
 6. The computing system of claim 1,wherein the process further comprises: determining which eye of the useris the user's dominant eye by determining how the user has positionedhimself or herself in relation to past stimulus; wherein the point atwhich the tip of the pyramid is placed is a point corresponding thedetermined dominant eye of the user.
 7. The computing system of claim 1,wherein the identifying multiple objects that at least partiallyintersect with at least part of the pyramid comprises limiting whichobjects are identified to only those that are on an opposite side of therectangle from the user.
 8. A computer-readable storage medium storinginstructions that, when executed by a computing system, cause thecomputing system to perform a process comprising: monitoring a fistcontrol point based on a position of a part of a first hand of the user;monitoring a second control point based on a position of a part of asecond hand of the user; defining a rectangle with opposite cornerscorresponding to the fist control point and the second control point;generating a pyramid projection, wherein a tip of the pyramid is at apoint corresponding to one of the user's eyes and each of the sides ofthe pyramid intersect with a different one of the lines that forms therectangle; identifying multiple objects that at least partiallyintersect with at least part of the pyramid projection, eithersimultaneously or at different times; and selecting the multipleobjects.
 9. The computer-readable storage medium of claim 8, wherein theprocess is performed in response to identifying that the user performeda gesture of: bringing a thumb-tip on the user's first hand togetherwith a fingertip on the user's first hand to make a first contact point;bringing a thumb-tip on the user's second hand together with a fingertipon the user's second hand to make a second contact point; bringing thefirst contact point together with the second contact point; and pullingthe first contact point away from the second contact point.
 10. Thecomputer-readable storage medium of claim 8, wherein the process furthercomprises: in response to the selecting the multiple objects, causing amenu to be attached to the first hand of the user; identifying aninteraction, by the user's second hand, with an item on the menu; and inresponse to the interaction, causing a resize or rotation of at leastone of the selected multiple objects.
 11. The computer-readable storagemedium of claim 8, wherein the process further comprises: monitoring arotation of the first hand of the user such that rotation of the firsthand of the user is mapped to rotation of at least one of the selectedmultiple objects; and based on the monitoring, detecting a rotation ofthe first hand of the user and, in response, causing a correspondingrotation of the at least one of the selected multiple objects.
 12. Thecomputer-readable storage medium of claim 8, wherein the process furthercomprises: determining which eye of the user is the user's dominant eyeby one or both of: determining which hand the user employs most often;and/or determining how the user has positioned himself or herself inrelation to past stimulus; wherein the point at which the tip of thepyramid is placed is a point corresponding the determined dominant eyeof the user.
 13. The computer-readable storage medium of claim 8,wherein the process further comprises: determining which eye of the useris the user's dominant eye by determining how the user has positionedhimself or herself in relation to past stimulus; wherein the point atwhich the tip of the pyramid is placed is a point corresponding thedetermined dominant eye of the user.
 14. The computer-readable storagemedium of claim 8, wherein the identifying multiple objects that atleast partially intersect with at least part of the pyramid compriseslimiting which objects are identified to only those that are on anopposite side of the rectangle from the user.
 15. A method comprising:monitoring a fist control point based on a position of a part of a firsthand of the user; monitoring a second control point based on a positionof a part of a second hand of the user; defining a rectangle withopposite corners corresponding to the fist control point and the secondcontrol point; generating a pyramid projection, wherein a tip of thepyramid is at a point corresponding to one of the user's eyes and eachof the sides of the pyramid intersect with a different one of the linesthat forms the rectangle; identifying multiple objects that at leastpartially intersect with at least part of the pyramid projection, eithersimultaneously or at different times; and selecting the multipleobjects.
 16. The method of claim 15, wherein the method is performed inresponse to identifying that the user performed a gesture of: bringing athumb-tip on the user's first hand together with a fingertip on theuser's first hand to make a first contact point; bringing a thumb-tip onthe user's second hand together with a fingertip on the user's secondhand to make a second contact point; bringing the first contact pointtogether with the second contact point; and pulling the first contactpoint away from the second contact point.
 17. The method of claim 15further comprising: in response to the selecting the multiple objects,causing a menu to be attached to the first hand of the user; identifyingan interaction, by the user's second hand, with an item on the menu; andin response to the interaction, causing a resize or rotation of at leastone of the selected multiple objects.
 18. The method of claim 15 furthercomprising: monitoring a rotation of the first hand of the user suchthat rotation of the first hand of the user is mapped to rotation of atleast one of the selected multiple objects; and based on the monitoring,detecting a rotation of the first hand of the user and, in response,causing a corresponding rotation of the at least one of the selectedmultiple objects.
 19. The method of claim 15 further comprising:determining which eye of the user is the user's dominant eye by one orboth of: determining which hand the user employs most often; and/ordetermining how the user has positioned himself or herself in relationto past stimulus; wherein the point at which the tip of the pyramid isplaced is a point corresponding the determined dominant eye of the user.20. The method of claim 15, wherein the identifying multiple objectsthat at least partially intersect with at least part of the pyramidcomprises limiting which objects are identified to only those that areon an opposite side of the rectangle from the user.